Solaris 10: tips and tricks for system administration

One of our users has asked me to help with PAM authentication for Apache 2 on one of our remote servers.

It did seem like a rather trivial task - download mod_auth_pam, compile it for Apache 2, make sure everything works. The remote server uses NIS, I could freely log onto it.

And you know what? It didn't work. The module compiled pretty easy, but I couldn't get it to work - error_log was reporting the same lines over and over again (I had to break the line to squeeze it into my blog format):



Up to this point, everything was done by the user himself with small help from me. At this stage is was decided I'd have a look myself. To make things easier, I also wanted to:

1) Move all the experiments onto my own Solaris 10 box
2) Make it all work for local users first, and only then get NIS working as well

Seeing the same errors on my box, I've decided to take a serious look at PAM. I've created Apache-related lines in /etc/pam.conf and added a debug option:



To make it more convenient to track all the messages, I've updated /etc/syslog.conf to include the following:



It got easier to see what was happening, at least I knew it was the /etc/pam.conf section I was thinking of. But the error was still there - my user definitely did exist in the system, only it wouldn't be recognized by Apache (and I've broken the lines yet again here):



It was the right time to look at how exactly PAM and Apache 2 interacted when I was giving the username and password trying to access my page. I immediately thought of Brendan Gregg's opensnoop script for DTrace. And it was this script indeed which helped me see the problem:



Apparently, httpd could not open /etc/shadow (-1 means error), obviously becase of permissions for the file. But just to make sure, I've looked for the error code (it's 13 as you can see):



So it all was suddenly clear, and I've proceeded like this:



All that's left to be done now was to make sure httpd.conf has a Group parameter set to this shadows group, and not nobody.

pS: you might ask why didn't it work for NIS on the remote server in first place. This is because /etc/nsswitch.conf had not this: passwd: files nis, but this: passwd: compat, so NIS maps were consulted, but only based on + and - specifications in /etc/passwd and /etc/shadow files. This means we still needed to be able to read /etc/shadow, even though later PAM would consult NIS.

Your next step in observing PHP with DTrace could easily be the following script. It times how much time (in nanoseconds) each Apache process has spent running a particular PHP function.

This script generates a table of Apache PIDs and PHP scripts started within, and later you'll get a table of all the functions with PIDs which ran them and the time spent.

You can download this script here: php-scripts-timer.d



A typical result of running this script while opening some PHP pages served by your Apache will look like this:

I've finally got some spare time to download a PEAR module for DTrace. What a great module! I've found a little problem with it, wrote a letter to Wez Furlong, and he immediately changed the source code to make it work even better. Thanks a lot, Wez! :)

So, here it is - my first PHP Dtrace script: php-scripts-snoop.d



So go ahead, download the script, do chmod a+x for it, and after you start it in a terminal of your web-server, you can run PHP scripts from this server, and our DTrace script will be reporting us whenever each PHP script stars or finishes, specifying which Apache process is responsible for this:



pS: yes, starting of today I've decided to provide download links for each of my scripts - should be more convenient for you now. This means that sooner or later I'll update all the archive blog entries to have download links for respective scripts.