CentOS Stream Upgrade, FreeIPA / KRB5 Login Fails

Support for security such as Firewalls and securing linux
Post Reply
tnsasse
Posts: 1
Joined: 2021/01/06 11:28:32

CentOS Stream Upgrade, FreeIPA / KRB5 Login Fails

Post by tnsasse » 2021/01/06 12:04:46

Hi folks,

I have recently started to upgrade a bunch of CentOS 8 servers to CentOS Stream 8. The servers are part of a FreeIPA Kerberos Domain. Since the upgrade/reboot I can no longer login via ssh + password with some users. Interestingly the same user can login with public key though. Wouldn't be a deal breaker, but login to IMAP on the mail servers is equally affected.

So I see the following in the log of the sssd (/var/log/sssd/krb5_child.log debug_level=8) on the client machine:

Login attempt with user tobi and password on server challenger with CentOS Stream 8 fails:

Code: Select all

(2021-01-06 12:19:01): [krb5_child[7027]] [main] (0x0400): krb5_child started.
(2021-01-06 12:19:01): [krb5_child[7027]] [unpack_buffer] (0x1000): total buffer size: [118]
(2021-01-06 12:19:01): [krb5_child[7027]] [unpack_buffer] (0x0100): cmd [241 (auth)] uid [1395000001] gid [1395000001] validate [true] enterprise principal [false] offline [false] UPN [tobi@REDACTED]
(2021-01-06 12:19:01): [krb5_child[7027]] [unpack_buffer] (0x0100): ccname: [KCM:] old_ccname: [KCM:] keytab: [/etc/krb5.keytab]
(2021-01-06 12:19:01): [krb5_child[7027]] [switch_creds] (0x0200): Switch user to [1395000001][1395000001].
(2021-01-06 12:19:01): [krb5_child[7027]] [sss_open_ccache_as_user] (0x0400): ccache KCM: is missing or empty
(2021-01-06 12:19:01): [krb5_child[7027]] [switch_creds] (0x0200): Switch user to [0][0].
(2021-01-06 12:19:01): [krb5_child[7027]] [old_ccache_valid] (0x0400): Saved ccache KCM: doesn't exist, ignoring
(2021-01-06 12:19:01): [krb5_child[7027]] [k5c_setup_fast] (0x0100): Fast principal is set to [host/challenger.REDACTED@REDACTED]
(2021-01-06 12:19:01): [krb5_child[7027]] [match_principal] (0x1000): Principal matched to the sample (host/challenger.REDACTED@REDACTED).
(2021-01-06 12:19:01): [krb5_child[7027]] [check_fast_ccache] (0x0200): FAST TGT is still valid.
(2021-01-06 12:19:01): [krb5_child[7027]] [become_user] (0x0200): Trying to become user [1395000001][1395000001].
(2021-01-06 12:19:01): [krb5_child[7027]] [main] (0x2000): Running as [1395000001][1395000001].
(2021-01-06 12:19:01): [krb5_child[7027]] [set_lifetime_options] (0x0100): No specific renewable lifetime requested.
(2021-01-06 12:19:01): [krb5_child[7027]] [set_lifetime_options] (0x0100): No specific lifetime requested.
(2021-01-06 12:19:01): [krb5_child[7027]] [set_canonicalize_option] (0x0100): Canonicalization is set to [true]
(2021-01-06 12:19:01): [krb5_child[7027]] [main] (0x0400): Will perform auth
(2021-01-06 12:19:01): [krb5_child[7027]] [main] (0x0400): Will perform online auth
(2021-01-06 12:19:01): [krb5_child[7027]] [tgt_req_child] (0x1000): Attempting to get a TGT
(2021-01-06 12:19:01): [krb5_child[7027]] [get_and_save_tgt] (0x0400): Attempting kinit for realm [REDACTED]
(2021-01-06 12:19:01): [krb5_child[7027]] [validate_tgt] (0x2000): Found keytab entry with the realm of the credential.
(2021-01-06 12:19:01): [krb5_child[7027]] [validate_tgt] (0x0400): TGT verified using key for [host/challenger.REDACTED@REDACTED].
(2021-01-06 12:19:01): [krb5_child[7027]] [get_and_save_tgt] (0x2000): Running as [1395000001][1395000001].
(2021-01-06 12:19:01): [krb5_child[7027]] [sss_get_ccache_name_for_principal] (0x2000): krb5_cc_cache_match failed: [-1765328191][Credentials cache I/O operation failed]
(2021-01-06 12:19:01): [krb5_child[7027]] [create_ccache] (0x0020): 995: [-1765328191][Credentials cache I/O operation failed]
(2021-01-06 12:19:01): [krb5_child[7027]] [map_krb5_error] (0x0020): 1838: [-1765328191][Credentials cache I/O operation failed]
(2021-01-06 12:19:01): [krb5_child[7027]] [k5c_send_data] (0x0200): Received error code 1432158209
(2021-01-06 12:19:01): [krb5_child[7027]] [pack_response_packet] (0x2000): response packet size: [4]
(2021-01-06 12:19:01): [krb5_child[7027]] [main] (0x0400): krb5_child completed successfully
Login with user john and password on server challenger with CentOS Stream 8 succeeds:

Code: Select all

(2021-01-06 12:38:19): [krb5_child[7490]] [main] (0x0400): krb5_child started.
(2021-01-06 12:38:19): [krb5_child[7490]] [unpack_buffer] (0x1000): total buffer size: [98]
(2021-01-06 12:38:19): [krb5_child[7490]] [unpack_buffer] (0x0100): cmd [249 (pre-auth)] uid [1395000014] gid [1395000014] validate [true] enterprise principal [false] offline [false] UPN [john@REDACTED]
(2021-01-06 12:38:19): [krb5_child[7490]] [unpack_buffer] (0x0100): ccname: [KCM:] old_ccname: [KCM:] keytab: [/etc/krb5.keytab]
(2021-01-06 12:38:19): [krb5_child[7490]] [k5c_setup_fast] (0x0100): Fast principal is set to [host/challenger.REDACTED@REDACTED]
(2021-01-06 12:38:19): [krb5_child[7490]] [match_principal] (0x1000): Principal matched to the sample (host/challenger.REDACTED@REDACTED).
(2021-01-06 12:38:19): [krb5_child[7490]] [check_fast_ccache] (0x0200): FAST TGT is still valid.
(2021-01-06 12:38:19): [krb5_child[7490]] [become_user] (0x0200): Trying to become user [1395000014][1395000014].
(2021-01-06 12:38:19): [krb5_child[7490]] [main] (0x2000): Running as [1395000014][1395000014].
(2021-01-06 12:38:19): [krb5_child[7490]] [set_lifetime_options] (0x0100): No specific renewable lifetime requested.
(2021-01-06 12:38:19): [krb5_child[7490]] [set_lifetime_options] (0x0100): No specific lifetime requested.
(2021-01-06 12:38:19): [krb5_child[7490]] [set_canonicalize_option] (0x0100): Canonicalization is set to [true]
(2021-01-06 12:38:19): [krb5_child[7490]] [main] (0x0400): Will perform pre-auth
(2021-01-06 12:38:19): [krb5_child[7490]] [tgt_req_child] (0x1000): Attempting to get a TGT
(2021-01-06 12:38:19): [krb5_child[7490]] [get_and_save_tgt] (0x0400): Attempting kinit for realm [REDACTED]
(2021-01-06 12:38:19): [krb5_child[7490]] [sss_krb5_prompter] (0x0200): Prompter interface isn't used for password prompts by SSSD.
(2021-01-06 12:38:19): [krb5_child[7490]] [get_and_save_tgt] (0x0400): krb5_get_init_creds_password returned [-1765328174] during pre-auth.
(2021-01-06 12:38:19): [krb5_child[7490]] [k5c_send_data] (0x0200): Received error code 0
(2021-01-06 12:38:19): [krb5_child[7490]] [pack_response_packet] (0x2000): response packet size: [12]
(2021-01-06 12:38:19): [krb5_child[7490]] [main] (0x0400): krb5_child completed successfully
(2021-01-06 12:38:22): [krb5_child[7491]] [main] (0x0400): krb5_child started.
(2021-01-06 12:38:22): [krb5_child[7491]] [unpack_buffer] (0x1000): total buffer size: [109]
(2021-01-06 12:38:22): [krb5_child[7491]] [unpack_buffer] (0x0100): cmd [241 (auth)] uid [1395000014] gid [1395000014] validate [true] enterprise principal [false] offline [false] UPN [john@REDACTED]
(2021-01-06 12:38:22): [krb5_child[7491]] [unpack_buffer] (0x0100): ccname: [KCM:] old_ccname: [KCM:] keytab: [/etc/krb5.keytab]
(2021-01-06 12:38:22): [krb5_child[7491]] [switch_creds] (0x0200): Switch user to [1395000014][1395000014].
(2021-01-06 12:38:22): [krb5_child[7491]] [switch_creds] (0x0200): Switch user to [0][0].
(2021-01-06 12:38:22): [krb5_child[7491]] [k5c_setup_fast] (0x0100): Fast principal is set to [host/challenger.REDACTED@REDACTED]
(2021-01-06 12:38:22): [krb5_child[7491]] [match_principal] (0x1000): Principal matched to the sample (host/challenger.REDACTED@REDACTED).
(2021-01-06 12:38:22): [krb5_child[7491]] [check_fast_ccache] (0x0200): FAST TGT is still valid.
(2021-01-06 12:38:22): [krb5_child[7491]] [become_user] (0x0200): Trying to become user [1395000014][1395000014].
(2021-01-06 12:38:22): [krb5_child[7491]] [main] (0x2000): Running as [1395000014][1395000014].
(2021-01-06 12:38:22): [krb5_child[7491]] [set_lifetime_options] (0x0100): No specific renewable lifetime requested.
(2021-01-06 12:38:22): [krb5_child[7491]] [set_lifetime_options] (0x0100): No specific lifetime requested.
(2021-01-06 12:38:22): [krb5_child[7491]] [set_canonicalize_option] (0x0100): Canonicalization is set to [true]
(2021-01-06 12:38:22): [krb5_child[7491]] [main] (0x0400): Will perform auth
(2021-01-06 12:38:22): [krb5_child[7491]] [main] (0x0400): Will perform online auth
(2021-01-06 12:38:22): [krb5_child[7491]] [tgt_req_child] (0x1000): Attempting to get a TGT
(2021-01-06 12:38:22): [krb5_child[7491]] [get_and_save_tgt] (0x0400): Attempting kinit for realm [HW.BYTE23.NET]
(2021-01-06 12:38:22): [krb5_child[7491]] [validate_tgt] (0x2000): Found keytab entry with the realm of the credential.
(2021-01-06 12:38:22): [krb5_child[7491]] [validate_tgt] (0x0400): TGT verified using key for [host/challenger.REDACTED@REDACTED].
(2021-01-06 12:38:22): [krb5_child[7491]] [get_and_save_tgt] (0x2000): Running as [1395000014][1395000014].
(2021-01-06 12:38:22): [krb5_child[7491]] [safe_remove_old_ccache_file] (0x0400): New and old ccache file are the same, none will be deleted.
(2021-01-06 12:38:22): [krb5_child[7491]] [k5c_send_data] (0x0200): Received error code 0
(2021-01-06 12:38:22): [krb5_child[7491]] [pack_response_packet] (0x2000): response packet size: [95]
(2021-01-06 12:38:22): [krb5_child[7491]] [main] (0x0400): krb5_child completed successfully
Login of user tobi on server discovery with CentOS 7.9.2009 and same password suceeds:

Code: Select all

(2021-01-06 12:43:04): [krb5_child[18185]] [main] (0x0400): krb5_child started.
(2021-01-06 12:43:04): [krb5_child[18185]] [unpack_buffer] (0x1000): total buffer size: [168]
(2021-01-06 12:43:04): [krb5_child[18185]] [unpack_buffer] (0x0100): cmd [241] uid [1395000001] gid [1395000001] validate [true] enterprise principal [false] offline [false] UPN [tobi@REDACTED]
(2021-01-06 12:43:04): [krb5_child[18185]] [unpack_buffer] (0x0100): ccname: [KEYRING:persistent:1395000001] old_ccname: [KEYRING:persistent:1395000001] keytab: [/etc/krb5.keytab]
(2021-01-06 12:43:04): [krb5_child[18185]] [switch_creds] (0x0200): Switch user to [1395000001][1395000001].
(2021-01-06 12:43:04): [krb5_child[18185]] [switch_creds] (0x0200): Switch user to [0][0].
(2021-01-06 12:43:04): [krb5_child[18185]] [k5c_setup_fast] (0x0100): Fast principal is set to [host/discovery.REDACTED@REDACTED]
(2021-01-06 12:43:04): [krb5_child[18185]] [match_principal] (0x1000): Principal matched to the sample (host/discovery.REDACTED@REDACTED).
(2021-01-06 12:43:04): [krb5_child[18185]] [check_fast_ccache] (0x0200): FAST TGT is still valid.
(2021-01-06 12:43:04): [krb5_child[18185]] [become_user] (0x0200): Trying to become user [1395000001][1395000001].
(2021-01-06 12:43:04): [krb5_child[18185]] [main] (0x2000): Running as [1395000001][1395000001].
(2021-01-06 12:43:04): [krb5_child[18185]] [set_lifetime_options] (0x0100): No specific renewable lifetime requested.
(2021-01-06 12:43:04): [krb5_child[18185]] [set_lifetime_options] (0x0100): No specific lifetime requested.
(2021-01-06 12:43:04): [krb5_child[18185]] [set_canonicalize_option] (0x0100): Canonicalization is set to [true]
(2021-01-06 12:43:04): [krb5_child[18185]] [main] (0x0400): Will perform online auth
(2021-01-06 12:43:04): [krb5_child[18185]] [tgt_req_child] (0x1000): Attempting to get a TGT
(2021-01-06 12:43:04): [krb5_child[18185]] [get_and_save_tgt] (0x0400): Attempting kinit for realm [REDACTED]
(2021-01-06 12:43:04): [krb5_child[18185]] [validate_tgt] (0x2000): Found keytab entry with the realm of the credential.
(2021-01-06 12:43:04): [krb5_child[18185]] [validate_tgt] (0x0400): TGT verified using key for [host/discovery.REDACTED@REDACTED].
(2021-01-06 12:43:04): [krb5_child[18185]] [get_and_save_tgt] (0x2000): Running as [1395000001][1395000001].
(2021-01-06 12:43:04): [krb5_child[18185]] [safe_remove_old_ccache_file] (0x0400): New and old ccache file are the same, none will be deleted.
(2021-01-06 12:43:04): [krb5_child[18185]] [k5c_send_data] (0x0200): Received error code 0
(2021-01-06 12:43:04): [krb5_child[18185]] [pack_response_packet] (0x2000): response packet size: [120]
(2021-01-06 12:43:04): [krb5_child[18185]] [main] (0x0400): krb5_child completed successfully
My first instinct was the password is incorrect: but if I provide an invalid password for user tobi on host challenger I end up with a different error message, other than that I can login to the FreeIPA UI, and other CentOS 7/8 hosts with the password just fine:

Code: Select all

(2021-01-06 12:59:03): [krb5_child[8126]] [become_user] (0x0200): Trying to become user [1395000001][1395000001].
(2021-01-06 12:59:03): [krb5_child[8126]] [main] (0x2000): Running as [1395000001][1395000001].
(2021-01-06 12:59:03): [krb5_child[8126]] [set_lifetime_options] (0x0100): No specific renewable lifetime requested.
(2021-01-06 12:59:03): [krb5_child[8126]] [set_lifetime_options] (0x0100): No specific lifetime requested.
(2021-01-06 12:59:03): [krb5_child[8126]] [set_canonicalize_option] (0x0100): Canonicalization is set to [true]
(2021-01-06 12:59:03): [krb5_child[8126]] [main] (0x0400): Will perform pre-auth
(2021-01-06 12:59:03): [krb5_child[8126]] [tgt_req_child] (0x1000): Attempting to get a TGT
(2021-01-06 12:59:03): [krb5_child[8126]] [get_and_save_tgt] (0x0400): Attempting kinit for realm [REDACTED]
(2021-01-06 12:59:03): [krb5_child[8126]] [sss_krb5_prompter] (0x0200): Prompter interface isn't used for password prompts by SSSD.
(2021-01-06 12:59:03): [krb5_child[8126]] [get_and_save_tgt] (0x0400): krb5_get_init_creds_password returned [-1765328174] during pre-auth.
(2021-01-06 12:59:03): [krb5_child[8126]] [k5c_send_data] (0x0200): Received error code 0
The FreeIPA server is version 4.6.8 on CentOS 7.9.2009. Help is much appreciated, as I am new to IPA/Kerberos.

Thanks,
Tobi

joob
Posts: 1
Joined: 2021/02/16 19:35:31

Re: CentOS Stream Upgrade, FreeIPA / KRB5 Login Fails

Post by joob » 2021/02/16 20:45:38

I noticed this problem as well. It seems to be due to expired kerberos ticket layout around.
You can try logging in with root + su to affected user + kdestroy expired ticket.

Then you can login fine and the problem is gone... until the ticket expires again. I added a "kdestroy" in my .bash_logout to work around the issue until it's fixed for good.

It first occured when updating from Centos 8 -> Centos Stream. Going back to C8 didn't help so I assume the update goofed up some configuration file.
I have other C8 (non stream) hosts where login works fine. Also F33 works fine.

-- john

Post Reply