ArticlesRocky Linux

Enabling Local User Login with Active Directory Authentication using Kerberos

Introduction

CIQ had a customer issue, where the customer was attempting to set up local users and then authenticate using Kerberos to their Active Directory server. The following guide will explain how this was resolved.

Problem

The customer was unable to authenticate via ssh to their Rocky Linux 9.2 LTS node using a local user account stored on the node. No passwords were locally saved in the /etc/shadow file and the passwords were only available on the Active Directory server.

The customer's configuration of sssd, Kerberos and Active Directory had worked without issue on CentOS 7.9, CentOS 8.1 and Rocky Linux 8.5. It was only on Rocky Linux 9.2 LTS where the customer was facing issues.

Symptoms

When attempting to log in to the server via ssh, the following authentication messages were observed under /var/log/secure:

Dec 9 11:07:31 <server_name> sshd[1155]: Received signal 15; terminating.
Dec 9 11:07:31 <server_name> sshd[79752]: Server listening on 0.0.0.0 port 22.
Dec 9 11:07:31 <server_name> sshd[79752]: Server listening on :: port 22.
Dec 9 11:10:04 <server_name> unix_chkpwd[79934]: check pass; user unknown
Dec 9 11:10:04 <server_name> unix_chkpwd[79934]: password check failed for user (<username>)
Dec 9 11:10:04 <server_name> sshd[79919]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<ip_address> user=<username>
Dec 9 11:10:44 <server_name> sshd[79919]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<ip_address> user=<username>
Dec 9 11:10:44 <server_name> sshd[79919]: pam_sss(sshd:auth): received for user <username>: 4 (System error)

Account retrieval errors could also be seen via the /var/log/sssd/sssd_pam.log and /var/log/sssd/sssd_nss.log files:

[root@<server_name> ~]# cat /var/log/sssd/sssd_pam.log
(2024-12-16 13:45:58): [pam] [server_setup] (0x3f7c0): Starting with debug level = 0x0070
(2024-12-16 13:47:20): [pam] [cache_req_common_process_dp_reply] (0x3f7c0): [CID#1] CR #3: Could not get account info [<account_number>]: SSSD is offline

[root@<server_name> ~]# cat /var/log/sssd/sssd_nss.log
(2024-12-16 13:45:58): [nss] [server_setup] (0x3f7c0): Starting with debug level = 0x0070
(2024-12-16 13:47:20): [nss] [cache_req_common_process_dp_reply] (0x3f7c0): [CID#5] CR #17: Could not get account info [<account_number>]: SSSD is offline

Resolution

The customer was utilizing id_provider = files in their /etc/sssd/sssd.conf file. Since the Rocky Linux 9.x series, this is a deprecated option in sssd.

Instead, they added the following settings into their sssd.conf file:

id_provider = proxy
krb5_server = example.com:88
proxy_lib_name = files

Then restarted the sssd service:

systemctl restart sssd`

The customer's sssd.conf file ultimately looked like the following (all sensitive customer information has been obfuscated):

[sssd]
    services = nss, pam
    domains = EXAMPLE.COM
    default_domain_suffix = example.com

[domain/EXAMPLE.COM]
    id_provider = proxy
    proxy_lib_name = files
    auth_provider = krb5
    krb5_realm = EXAMPLE.COM
    krb5_server = example.com:88
    krb5_validate = false
    krb5_ccachedir = /var/tmp
    krb5_keytab = /etc/krb5.keytab

[nss]
filter_groups = root,<local user>
filter_users = root,<local user>

In addition, the customer's /etc/kr5b.conf file:

includedir /etc/krb5.conf.d/

[logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

[libdefaults]
    default_realm=EXAMPLE.COM
    dns_lookup_realm = true
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    default_ccache_name = KEYRING:persistent:%{uid}

[realms]
     EXAMPLE.COM = {
     kdc = example.com:88
     admin_server = example.com:749
     }
     example.com = {
     kdc = example.com:88
     admin_server = example.com:749
     }

[domain_realm]
      example.com = EXAMPLE.COM
      example.com = EXAMPLE.COM

[appdefaults]
 pam = {
   debug = true
   ticket_lifetime = 36000
   renew_lifetime = 36000
   forwardable = true
   krb4_convert = false
 }

Furthermore, the customer's /etc/nsswitch.conf file:

# In order of likelihood of use to accelerate lookup.
shadow:     files
hosts:      files dns myhostname

aliases:    files
ethers:     files
gshadow:    files
# Allow initgroups to default to the setting for group.
# initgroups: files
networks:   files dns
protocols:  files
publickey:  files
rpc:        files

Troubleshooting notes

Adding debug_level = 9 to each section of the sssd.conf file, yields more verbose logs under /var/log/sssd/*

When troubleshooting the issue, the following logs are essential:

/var/log/sssd/*
/var/log/secure

In addition, these files are also important to check:

/etc/kr5b.conf
/etc/nsswitch.conf
/etc/pam.d/password-auth
/etc/pam.d/system-auth
/etc/sssd/sssd.conf

Next are these command outputs to run:

getent passwd <username>
klist
kinit <username>
ssh -o PubkeyAuthentication=no -vvv -k <username>@<server_name>

Finally, generating an sosreport from the node.

References & related articles

Mapping local users to Kerberos principals with SSSD

sssd.conf man page

sssd-files man page

sssd-krb5 man page