Centos 7 and Identity Management Slow login

seli · Post by **seli** » 2015/08/06 10:27:15

Hi everybody,
I arrived in an academic lab few time after a (not so) sysadmin left.
I inherited a server running centos 7 and Identity Management (Kerberos, 389DS, ...) with NFS.
Everything concerning login is very slow (seconds)
I already solved a lot of problems on this server, but I am neither a sysadmin nor a linux guru and I don't know where and what to look for ?
Kerberos ? 389DS ? NFS ? SElinux ? sssd ? ...
I will provide you whatever is needed (logs, output ...)

Thanks for your help
A desperate bioinformatician

aks · Post by **aks** » 2015/08/06 17:16:59

I'd guess that it's a networking problem, given you have centralized login. Perhaps try and login as a local user (i.e.: not a Kerberos/LDAP) who does not mount a directory over NFS. If that's quick, then it's probably the time it takes to do all the networking stuff or a problem server side.

seli · Post by **seli** » 2015/08/07 08:23:25

Hi aks,

Thanks for your reply, as I have said, I solved a lot of problems, including NTP and DNS (bind), my first thoughts concerning this problem. I've already checked firewall and SElinux. Networking between clients and server is really fast when using local accounts.
I rebooted the server yesterday, and looking the logs I've seen this:
journalctl:
krb5kdc[2973]: krb5kdc: cannot initialize realm BIOINF.LOCAL - see log file for details
systemd[1]: krb5kdc.service: control process exited, code=exited status=1
systemd[1]: Failed to start Kerberos 5 KDC.
systemd[1]: Unit krb5kdc.service entered failed state.

krb5kdc.log:
krb5kdc: Server error - while fetching master key K/M for realm BIOINF.LOCAL
krb5kdc[7835](Error): preauth pkinit failed to initialize: No realms configured correctly for pkinit support

Any idea ?

aks · Post by **aks** » 2015/08/07 16:46:11

PKINIT is a preauthentication mechanism for Kerberos 5 which uses X.509 certificates to authenticate the KDC to clients and vice versa. PKINIT can also be used to enable anonymity support, allowing clients to communicate securely with the KDC or with application servers without authenticating as a particular client principal.

source: http://web.mit.edu/kerberos/krb5-latest ... kinit.html

That usually means means the KDC can't contact the certificate or LDAP server (I think FreeIPA stores that stuff in the LDAP server) - so the Kerberos servers can't "talk" to the 389 server in some manner. The could be ports (i.e.: network connectivity), the LDAP server is not available (is it running?) or the DNS entries are missing. Perhaps the X509 certificates have expired or the CA is not trusted (if it's self signed)?

Or perhaps your Kerberos configuration doesn't contain the necessary pkinit_identity (points to the server certificate in pem format), pkinit_anchors (points to the ca certificate in pem format), and kdc_tcp_ports options.

Some (non CentOS specific information, just for a bit of background): http://k5wiki.kerberos.org/wiki/Pkinit_ ... ring_a_KDC and http://web.mit.edu/kerberos/krb5-latest ... kinit.html)

That's the kind of thing that will cause latency.

seli · Post by **seli** » 2015/08/10 09:48:01

The DNS entries seems ok:

Code: Select all

$TTL 3600	; 1 hour
@		IN 	SOA	lead.bioinf.local. root.bioinf.local. (
				2015063003 ; serial
				900        ; refresh (15 minutes)
				600        ; retry (10 minutes)
				86400      ; expire (1 day)
				3600       ; minimum (1 hour)
				)	   ;
; DNS Servers
			IN  NS		lead.bioinf.local.
			IN  NS		lead1.bioinf.local.
; ldap servers
_ldap._tcp		IN SRV 0 100 389	lead.bioinf.local.
; kerberos realm
_kerberos		IN TXT BIOINF.LOCAL
; kerberos servers
_kerberos._tcp		IN SRV 0 100 88		lead.bioinf.local.
_kerberos._udp		IN SRV 0 100 88		lead.bioinf.local.
_kerberos-master._tcp	IN SRV 0 100 88		lead.bioinf.local.
_kerberos-master._udp	IN SRV 0 100 88		lead.bioinf.local.
_kpasswd._tcp		IN SRV 0 100 464	lead.bioinf.local.
_kpasswd._udp		IN SRV 0 100 464	lead.bioinf.local.
; ntp server
_ntp._udp 		IN SRV 0 100 123	lead.bioinf.local.
; CNAME for IPA CA replicas (used for CRL, OCSP)
ipa-ca			IN A			192.168.0.3
; servers
lead1			IN A		192.168.0.2
lead			IN A		192.168.0.3
...

The ldap server seems to be running:

Code: Select all

[root@lead named]# systemctl status dirsrv@BIOINF-LOCAL.service 
dirsrv@BIOINF-LOCAL.service - 389 Directory Server BIOINF-LOCAL.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled)
   Active: active (running) since Mon 2015-08-10 10:20:03 CEST; 57min ago
  Process: 18861 ExecStart=/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-%i -i /var/run/dirsrv/slapd-%i.pid -w /var/run/dirsrv/slapd-%i.startpid (code=exited, status=0/SUCCESS)
 Main PID: 18862 (ns-slapd)
   CGroup: /system.slice/system-dirsrv.slice/dirsrv@BIOINF-LOCAL.service
           └─18862 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-BIOINF-LOCAL -i /va...

Aug 10 11:11:47 lead.bioinf.local ns-slapd[18862]: GSSAPI server step 3
Aug 10 11:11:49 lead.bioinf.local ns-slapd[18862]: GSSAPI server step 1
Aug 10 11:11:50 lead.bioinf.local ns-slapd[18862]: GSSAPI server step 2
Aug 10 11:11:50 lead.bioinf.local ns-slapd[18862]: GSSAPI server step 3
Aug 10 11:13:06 lead.bioinf.local ns-slapd[18862]: GSSAPI server step 1
Aug 10 11:13:07 lead.bioinf.local ns-slapd[18862]: GSSAPI server step 2
Aug 10 11:13:07 lead.bioinf.local ns-slapd[18862]: GSSAPI server step 3
Aug 10 11:14:25 lead.bioinf.local ns-slapd[18862]: GSSAPI server step 1
Aug 10 11:14:26 lead.bioinf.local ns-slapd[18862]: GSSAPI server step 2
Aug 10 11:14:26 lead.bioinf.local ns-slapd[18862]: GSSAPI server step 3

Code: Select all

[test@argon ~]$ ipa user-find test
--------------
1 user matched
--------------
  User login: test
  First name: test
  Last name: user
  Home directory: /home/test
  Login shell: /bin/bash
  Email address: test@bioinf.local
  UID: 1713400050
  GID: 1713400050
  Account disabled: False
  Password: True
  Kerberos keys available: True
----------------------------
Number of entries returned 1
----------------------------

Firewall ports are open:

Code: Select all

firewall-cmd --get-default-zone
work

firewall-cmd --zone=work --list-ports
80/tcp 464/tcp 88/udp 389/tcp 53/tcp 662/udp 892/tcp 749/udp 111/udp 20048/udp 7389/tcp 9445/tcp 9946/udp 2049/tcp 749/tcp 892/udp 123/udp  111/tcp 636/tcp 443/tcp 464/udp 88/tcp 662/tcp

Perhaps the X509 certificates have expired or the CA is not trusted (if it's self signed)?

How to test that ?

Thanks a lot for your time !

aks · Post by **aks** » 2015/08/10 17:06:46

You can use OpenSSL "client-mode" to get the certificate and to verify (http://stackoverflow.com/questions/7084 ... om-openssl).
In the past I've used an ldapclient (gq_ldap I think but there are many to choose from).

seli · Post by **seli** » 2015/08/17 09:03:05

I added "selinux_provider = none" to the sssd.conf and then restarted the service .... and it seems to solve the problem (almost) !!! Logins are near as fast as when using local users.
But I have now to understand why is it complaining (sssd_domain.log) :

Code: Select all

(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_get_primary_name] (0x0400): Processing object test
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_save_user] (0x0400): Processing user test
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_idmap_domain_has_algorithmic_mapping] (0x0080): Could not parse domain SID from [(null)]
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_save_user] (0x0400): Adding original memberOf attributes to [test].
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_save_user] (0x0400): Adding user principal [test@BIOINF.LOCAL] to attributes of [test].
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_save_user] (0x0400): Storing info for user test

Code: Select all

(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_get_primary_name] (0x0400): Processing object test
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_save_group] (0x0400): Processing group test
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_idmap_domain_has_algorithmic_mapping] (0x0080): Could not parse domain SID from [(null)]
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_process_ghost_members] (0x0400): The group has 0 members
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_process_ghost_members] (0x0400): Group has 0 members
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_save_group] (0x0400): Storing info for group test
...
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_get_primary_name] (0x0400): Processing object test
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_save_grpmem] (0x0400): Processing group test
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_save_grpmem] (0x0400): Failed to get group sid
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_save_grpmem] (0x0400): No members for group [test]
Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_get_primary_name] (0x0400): Processing object test
...
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_save_group] (0x0400): Processing group test
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_idmap_domain_has_algorithmic_mapping] (0x0080): Could not parse domain SID from [(null)]
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_process_ghost_members] (0x0400): Group has 0 members
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [sdap_save_group] (0x0400): Storing info for group test
(Fri Aug 14 09:25:59 2015) [sssd[be[bioinf.local]]] [enum_groups_done] (0x0100): Groups higher USN value: [318860]

CentOS

Centos 7 and Identity Management Slow login

Centos 7 and Identity Management Slow login

Re: Centos 7 and Identity Management Slow login

Re: Centos 7 and Identity Management Slow login

Re: Centos 7 and Identity Management Slow login

Re: Centos 7 and Identity Management Slow login

Re: Centos 7 and Identity Management Slow login

Re: Centos 7 and Identity Management Slow login