DTrace: mod_auth_pam and apache2

One of our users has asked me to help with PAM authentication for Apache 2 on one of our remote servers.

Note: going with mod_auth_pam was decision made without me, and I was only asked to help get the chosen solution work. This being said, the recommended way to do external authentication for Apache 2 now is to use the mod_authz_external module and pwauth password authenticator.

It did seem like a rather trivial task – download mod_auth_pam, compile it for Apache 2, make sure everything works. The remote server uses NIS, I could freely log onto it.

And you know what? It didn’t work!

The module compiled pretty easily, but I couldn’t get it to work – error_log was reporting the same lines over and over again (I had to break the line to squeeze it into my blog format):

[Tue Dec 20 16:48:44 2005] [error] [client xxx.xxx.xxx.xxx]
PAM: user 'greys' - not authenticated: No account present for user

Up to this point, everything was done by the user himself with small help from me. At this stage is was decided I’d have a look myself. To make things easier, I also wanted to:

1. Move all the experiments onto my own Solaris 10 box
2. Make it all work for local users first, and only then get NIS working as well

Seeing the same errors on my box, I’ve decided to take a serious look at PAM. I’ve created Apache-related lines in /etc/pam.conf and added a debug option:

httpd  auth required           pam_authtok_get.so.1 debug
httpd  auth required           pam_unix_auth.so.1 debug

To make it more convenient to track all the messages, I’ve updated /etc/syslog.conf to include the following:

auth.debug                      /var/log/auth_log
user.debug                      /var/log/user_log

It got easier to see what was happening, at least I knew it was the /etc/pam.conf section I was thinking of. But the error was still there – my user definitely did exist in the system, only it wouldn’t be recognized by Apache (and I’ve broken the lines yet again here):

Dec 21 09:47:35 pele httpd[29143]: [ID 634615 user.debug]
pam_authtok_get:pam_sm_authenticate: flags = 1
Dec 21 09:47:35 pele httpd[29143]: [ID 378613 user.debug]
pam_dhkeys: user greys not found
Dec 21 09:47:35 pele httpd[29143]: [ID 896952 user.debug]
pam_unix_auth: entering pam_sm_authenticate()
Dec 21 09:47:35 pele httpd[29143]: [ID 219349 user.debug]
pam_unix_auth: user greys not found

It was the right time to look at how exactly PAM and Apache 2 interacted when I was giving the username and password trying to access my page. I immediately thought of Brendan Gregg’s OpenSnoop script for DTrace. And it was this script indeed which helped me see the problem:

60001 24744 httpd         -1   2 /etc/pam_debug
60001 24744 httpd         11   0 /etc/pam.conf
60001 24744 httpd         11   0 /usr/lib/security/pam_authtok_get.so.1
60001 24744 httpd         11   0 /usr/lib/passwdutil.so.1
60001 24744 httpd         11   0 /usr/lib/libsldap.so.1
60001 24744 httpd         11   0 /usr/lib/security/pam_dhkeys.so.1
60001 24744 httpd         11   0 /usr/lib/security/pam_unix_cred.so.1
60001 24744 httpd         11   0 /lib/libbsm.so.1
60001 24744 httpd         11   0 /lib/libsecdb.so.1
60001 24744 httpd         11   0 /usr/lib/libproject.so.1
60001 24744 httpd         11   0 /lib/libproc.so.1
60001 24744 httpd         11   0 /lib/librtld_db.so.1
60001 24744 httpd         11   0 /lib/libelf.so.1
60001 24744 httpd         11   0 /lib/libctf.so.1
60001 24744 httpd         11   0 /usr/lib/security/pam_unix_auth.so.1
60001 24744 httpd         -1   2 /var/run/syslog_door
60001 24744 httpd         -1  13 /etc/shadow
60001 24744 httpd         11   0 /var/run/name_service_door
60001 24744 httpd         11   0 /etc/passwd
60001 24744 httpd         -1  13 /etc/shadow
60001 24744 httpd         -1   2 /var/run/syslog_door

Apparently, httpd could not open /etc/shadow (-1 means error), obviously becase of permissions for the file. But just to make sure, I’ve looked for the error code (it’s 13 as you can see):

solaris# grep 13 /usr/include/sys/errno.h
#define EACCES  13      /* Permission denied                    */

So it all was suddenly clear, and I’ve proceeded like this:

solaris# groupadd shadows
solaris# chown root:shadows /etc/shadow
solaris# chmod 440 /etc/shadow

All that’s left to be done now was to make sure httpd.conf has a Group parameter set to this shadows group, and not nobody.

pS: you might ask why didn’t it work for NIS on the remote server in first place. This is because /etc/nsswitch.conf had not this:

passwd: files nis

but this:

passwd: compat

so NIS maps were consulted, but only based on + and – specifications in /etc/passwd and /etc/shadow files. This means we still needed to be able to read /etc/shadow, even though later PAM would consult NIS.