publickey doesn't work when sshd is run from systemd

[zoltan]

Funny thing...I can set up .ssh/authorized_keys and can log in without password BUT ONLY if I first: systemctl stop sshd ; /usr/sbin/sshd

If sshd the service is run from systemd, i.e., killall sshd ; systemctl start sshd [to continue from above example], user must then enter password (note that I've already 'PasswordAuthentication yes' in /etc/ssh/sshd_config).

So...just curious...what does systemd do to my beloved publickey? I've checked $OPTIONS, etc. for how sshd is run as a service, can't see anything obvious, the logs are useless, and even client-side -v provides nothing extra to go on. This is on CentOS-7. I've done both RSA and DSA, and both work running /usr/sbin/sshd, but neither work under 'systemctl start sshd'. No other changes that I can see.

Any thoughts? Thanks...

[TrevorH]

Run restorecon -r ~/.ssh

[zoltan]

Thanks for the reply...I tried it, though I didn't have high hopes for it...and no, it did not work

Why do you expect that SELinux context would be different when sshd is run from inside the systemd frame-work vs. just being run as /usr/sbin/sshd? I wouldn't expect that, and the restorecon -r ~/.ssh had no effect. I can still login with publickey when I run /usr/sbin/sshd directly, but not when I run it as 'systemctl start sshd'.

Anybody else care to take a swing at it?

[aks]

Just to recap (and to confirm I've got everything right) - when sshd is running from boot (started via systemd) the passwordless ssh facility does NOT work, but if you manually start sshd (as root) then the passwordless ssh facility works. Is that right? If it is that strongly suggests sshd is not running as root (when started by systemd) and can not read the "passwordless files". So how to go about troubleshooting, start by checking the user sshd is running as when started via systemd (I thuink it's supposed to be root, but that's from memory). Then check what context (SELinux context) the daemon is running as, then check the authorized_keys to see who can read the file and check it's SELinux context. Okay so how to go about this:

1) From boot type ps -eZf | grep sshd # will tell you who sshd is running as and it's SELinux context.
2) ls -lhdZ $HOME/.ssh # will tell you the permissions and SELinux context of the $HOME/.ssh directory.
3) ls -lhZ $HOME/.ssh # will tell you the SELinux context and permissions of the authorized_keys file.

But you probably already knew this. Also remember sshd is really anal about permissions on .ssh and authorized_keys. Also have a look in /var/log for the log files (CentOS 7's sshd many use systemd's log file for messages, I'm not sure, so check also systemd's logfile).

Perhaps post back the results of the above if it doesn't help.

So perhaps Trevor's idea wasn't so bad after all.

[TrevorH]

Things started by the init subsystem on CentOS of any variety have a different selinux context than those same things started outside it. If something works when started by hand and not via init/systemd then that almost always implies an selinux problem.

[zoltan]

Okay, so...I've done some further digging...and found a part of the problem that I had not described before, and there's the rub. As in many environments, user's home directories are on an NFS-mounted fs. Given corporate environments, root access to NFS mounts is restricted. Root cannot even read a user's ~/.ssh/ directory, much less anything in it.

So it is a permission issue, but opposite of what you were thinking. In this case root _cannot_ read ~/.ssh, and adding the -d option to the sshd.service file confirmed this: got permission denied when it tried to read authorized_keys file. And knowing this, seems like expected behavior.

So the issue now becomes not "why does publickey auth. fail when sshd run as systemd service," but rather "why does public key succeed when /usr/sbin/sshd is directly"? I mean root is root. I'm still running /usr/sbin/sshd as root. So somewhere along the way, that drops privilege and consults .ssh/authorized_keys as the incoming UID instead of root is the only thing I can think of. I'm not sure how to confirm that outside of the fact /usr/sbin/sshd -d -D does show a successful read of authorized keys file and a passwordless login works. So let's assume that the priv. drop occurs and that's why this works.

Then the question is why and/or how does systemd affect the priv-drop sequence of sshd? Command-line options are the same all around. Its something inside systemd...

BTW, I have a usable work-around in setting "AuthorizedKeysFile /home/%u/.ssh/authorized_keys" and setting up shop there. So in case anyone runs into this..that works (/home is local fs). But I'm still curious about this systemd beast...

[jyoung]

If you manually call sshd at the command line, it'll inherit the context of the user which issued the startup. As root, on my system, I produce this:

Code: Select all

[root@young jyoung]# pgrep -f sshd
[root@young jyoung]# /usr/sbin/sshd
[root@young jyoung]# pgrep -fal sshd
15552 /usr/sbin/sshd
[root@young jyoung]# ps -auxZ | grep sshd | grep -v grep
staff_u:system_r:unconfined_t:s0-s0:c0.c1023 root 15552 0.0  0.0 82824 2888 ?  Ss   08:57   0:00 /usr/sbin/sshd

However, if I use systemd to start sshd, the context is running with a different context:

Code: Select all

[root@young jyoung]# pgrep -fal sshd
[root@young jyoung]# systemctl start sshd
[root@young jyoung]# pgrep -fal sshd
15571 /usr/sbin/sshd -D
[root@young jyoung]# ps -auxZ | grep sshd | grep -v grep
system_u:system_r:sshd_t:s0-s0:c0.c1023 root 15571 0.0  0.0 82824 6080 ?       Ss   08:58   0:00 /usr/sbin/sshd -D

Turn on the boolean use_nfs_home_dirs, and you won't have this issue anymore. As Trevor suggested, this is related to SELinux.

Code: Select all

[root@young jyoung]# sesearch -A -C -s sshd_t -t nfs_t
Found 7 semantic av rules:
   allow sshd_t filesystem_type : filesystem getattr ; 
   allow login_pgm exec_type : file getattr ; 
   allow sshd_t file_type : filesystem getattr ; 
   allow sshd_t file_type : dir { getattr search open } ; 
DT allow userdom_home_manager_type nfs_t : file { ioctl read write create getattr setattr lock append unlink link rename open } ; [ use_nfs_home_dirs ]
DT allow userdom_home_manager_type nfs_t : dir { ioctl read write create getattr setattr lock unlink link rename add_name remove_name reparent search rmdir open } ; [ use_nfs_home_dirs ]
DT allow userdom_home_manager_type nfs_t : lnk_file { ioctl read write create getattr setattr lock append unlink link rename } ; [ use_nfs_home_dirs ]

[zoltan]

Thanks everyone for your input. Appreciate the pointers on SELinux. Its not my idea to have it enabled on our boxes in the first place, but--lucky me--I get to go learn all about it now. Thanks Again.

[scaine]

I had the same problem described above.
However, I arrived at the answer a different way (using audit2why) and I thought this method may be useful for people to know.
audit2why is provided by the package policycoreutils-python-2.2.5-20.el7.x86_64

Code: Select all

yum install policycoreutils-python-2.2.5-20.el7.x86_64

to use the command i did the following

Code: Select all

grep ssh /var/log/audit/audit.log | audit2why -w

which not only tells you what is the problem, but also tells you the exact command you need to run to fix it!
which of course is

Code: Select all

setsebool -P use_nfs_home_dirs 1