
## Introduction

This guide will have you setup and run Ascender on a K3s cluster.

## Problem

Many customers face difficulties when managing their infrastructure at scale. This is where Ascender comes in to ensure scalable patching, security hardening and more across your fleet.

## Symptoms

A new user may encounter difficulties installing Ascender onto their K3s cluster.

## Resolution

### Prerequisites

* Same OS family as Rocky Linux. Rocky Linux 8.x or Rocky Linux 9.x is recommended as the base OS.

* System requirements:

  * CPUs: 2

  * Memory: 8GB (if installing both Ascender and Ledger)

  * Disk: 20GB (if installing both Ascender and Ledger)

* Internet access for the K3s cluster.

* Perform the below steps with a user that has privilege escalation. Do not use the `root` user account.

### Setup

* Update the node running K3s:

```bash
sudo dnf update -y
```

* Install `git`:

```bash
sudo dnf install -y git
```

* Allow these ports in `firewalld`:

```bash
sudo firewall-cmd --permanent --zone=public --add-port=22/tcp
sudo firewall-cmd --permanent --zone=public --add-port=80/tcp
sudo firewall-cmd --permanent --zone=public --add-port=443/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5432/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5895/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5986/tcp
sudo firewall-cmd --reload
```

* Clone the Ascender repository:

```bash
git clone https://github.com/ctrliq/ascender-install.git
```

* Navigate to the installation directory:

```bash
cd ascender-install
```

* Run the `config_vars.sh` script:

```bash
./config_vars.sh
```

* The script will present you with multiple questions. Answer each of them according to how you want to set up your Ascender environment and if you already have K3s installed on your node.

* Run the setup script to install Ascender:

```bash
sudo ./setup.sh
```

* Once complete, the final playbook recap will look like the following example:

```bash
PLAY RECAP *************************************************************************************************************************
ascender_host              : ok=14   changed=6    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0
localhost                  : ok=72   changed=27   unreachable=0    failed=0    skipped=4    rescued=0    ignored=0

ASCENDER SUCCESSFULLY SETUP
```

### Accessing Ascender

* You can access Ascender directly via the internal CLUSTER IP address.

## Notes

* To monitor the Ascender setup on your K3s cluster during install:

```bash
kubectl get pod -n ascender -w
```

* For further details surrounding particular pods:

```bash
kubectl describe pod -n ascender [pod name]
```

## References & related articles

[Ascender General Prerequisites](https://github.com/ctrliq/ascender-install/tree/main?tab=readme-ov-file#general-prerequisites)
[Ascender K3s Install Guide](https://github.com/ctrliq/ascender-install/blob/main/docs/k3s/README.md)


Quick Start Guide for Installing Ascender on K3s


## Introduction

Customers often need a custom ISO for a litany of reasons - perhaps they require a mirror of a repository or to edit kernel cmdline parameters for their own customers.

## Problem

This guide was created after a customer raised a question in [#1629 - creating a custom installation ISO](https://ciq.zendesk.com/agent/tickets/1629), asking for assistance on how to create a Rocky Linux 8.10 ISO. 

## Symptoms

The customer did not have clear direction or resources on how to build their own ISO.

## Resolution

* The customer was given multiple documentation sources to reference, a high level ISO generation guide and multiple commands to execute.
* A guide on how to create a custom Rocky Linux ISO was also pushed upstream to the Rocky Linux Docs. Please find the same guide below:

### Prerequisites

* A 64 bit machine running Rocky Linux 9 to build the new ISO image.
* A Rocky Linux 9 DVD ISO image.
* A `kickstart` file to apply to the ISO.
* Read the Lorax [Quickstart](https://weldr.io/lorax/lorax.html#quickstart) and [mkksiso](https://weldr.io/lorax/mkksiso.html) documentation to become familiar with how the `Anaconda` `boot.iso` is created.

### Package installation and setup

* Install the `lorax` package:
```
sudo dnf install -y lorax
```

### Building the ISO with a kickstart file

* Run the `mkksiso` command to add a `kickstart` file and then build a new ISO:
```
mkksiso --ks <PATH_TO_KICKSTART_FILE> <PATH_TO_ISO_TO_MODIFY> <OUTPUT_PATH_FOR_BUILT_ISO>
```
* Below is an example `kickstart` file `example-ks.cfg`, which sets up a Rocky Linux 9.5 `Server With GUI` environment:
```
lang en_GB
keyboard --xlayouts='us'
timezone Asia/Tokyo --utc
reboot
cdrom
bootloader --append="rhgb quiet crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
zerombr
clearpart --all --initlabel
autopart
network --bootproto=dhcp
firstboot --disable
selinux --enforcing
firewall --enabled
%packages
@^server-product-environment
%end
```

### Adding a repository with its packages to an ISO image

* Make sure the repository you want to add has the `repodata` directory inside of it. If not, this can be created using the `createrepo_c` command and this can be installed with `sudo dnf install -y createrepo_c`
* Add the repository to your `kickstart` file, using the following syntax:
```
repo --name=extra-repo --baseurl=file:///run/install/repo/<YOUR_REPOSITORY>/
```
* Add your repository using the `--add` flag with the `mkksiso` tool:
```
mkksiso --add <LINK_TO_YOUR_REPOSITORY> --ks <PATH_TO_KICKSTART_FILE> <PATH_TO_ISO_TO_MODIFY> <OUTPUT_PATH_FOR_BUILT_ISO>
```

* The process is detailed further using the `baseos` repository in the example below:
* The `base os` repository will be locally downloaded along with all of its packages:
```
dnf reposync -p ~ --download-metadata --repo=baseos
```
* Then add the repository to the `kickstart` file:
```
repo --name=extra-repo --baseurl=file:///run/install/repo/baseos/
```
* The `kickstart` file would look like the following:
```
lang en_GB
keyboard --xlayouts='us'
timezone Asia/Tokyo --utc
reboot
cdrom
bootloader --append="rhgb quiet crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
zerombr
clearpart --all --initlabel
autopart
network --bootproto=dhcp
firstboot --disable
selinux --enforcing
firewall --enabled
%packages
@^server-product-environment
repo --name=extra-repo --baseurl=file:///run/install/repo/baseos/
%end
```
* Then point the `mkkiso` command directly to the repository directory and build the ISO:
```
mkksiso --add ~/baseos --ks example-ks.cfg ~/Rocky-9.5-x86_64-dvd.iso ~/Rocky-9.5-x86_64-dvd-new.iso
```

### Conclusion

What has been discussed here are just a few of the options available to tweak and build your own Rocky Linux ISO. For further ways, including modifying the kernel cmdline arguments, the author highly recommends to go through the [mkksiso](https://weldr.io/lorax/mkksiso.html) documentation in more detail.


## References & related articles

[Anaconda Documentation](https://anaconda-installer.readthedocs.io/en/latest/)\
[Automating the Installation with Kickstart - Fedora Wiki](https://docs.fedoraproject.org/en-US/fedora/f36/install-guide/advanced/Kickstart_Installations/#chap-kickstart-installations)\
[Dracut - Fedora Wiki](https://fedoraproject.org/wiki/Dracut)\
[Lorax Documentation](https://weldr.io/lorax/)\
[mkksiso Documentation](https://weldr.io/lorax/mkksiso.html)\
[Mock Repository](https://github.com/rpm-software-management/mock)\
[Peridot Repository](https://github.com/rocky-linux/peridot)


Creating a Custom Rocky Linux ISO


## Introduction

This article is a great reference for customers looking to learn more about how CIQ evaluates and remediates security vulnerabilities. 

## FAQ 

**What CVE’s are addressed by CIQ Bridge and CIQ LTS?**  
For both Bridge and LTS subscriptions, CIQ focuses on resolving CVEs with CVSS scores of 7 and above. CIQ also considers customer impact and environmental factors.CIQ evaluates CVE’s taking into account the OS, the build environment and other Linux vendor scores to determine the applicability of the CVE to our distribution. CIQ prioritizes CVE’s based on CVSS score, package exposure, popularity, confidence in the potential fixes (vs. risk of them introducing bugs), and customer requests.  

**What does the CVSS Score mean?**  
CIQ evaluates scores from [NIST CVSS 3.x Scoring System](https://nvd.nist.gov/vuln-metrics/cvss), MITRE and other OS Vendors to evaluate and rank vulnerabilities in a standardized and repeatable way.  These scores are used as inputs for CIQ to determine the appropriate impact of the CVE to our distribution.

**Are there SLA’s for CVE remediation?**  
CIQ does not provide SLA’s for any CVE remediation. They are prioritized based on the criteria discussed above.

**Where can I see the status of a given CVE?**  
All fixes can be found in the CSAF files provided by the public [CIQ Advisories Repository](https://github.com/ctrliq/advisories). CVE fixes are also published in the [Mountain Portal](https://secure.ciq.com/).  For any other status updates, please reach out to your Customer Success Manager or open a Support ticket.

**How do CSAF files relate to the output of my scanning tool?**  
Scanners use CSAF files and data from OS vendors to determine if a CVE is fixed based on the version of the RPM. Scanner vendors are responsible for processing Rocky Linux and CIQ CSAF files and advisories to determine if a fix has been released for a given CVE.  If your scanner is not ingesting the CSAF files from Rocky/CIQ LTS it will produce false positives.

**How do CVE’s relate to Security Advisories?**  
Security advisories can contain one or more CVE’s. CIQ evaluates CVEs individually and not Security Advisories. CVE’s will be fixed based on the criteria mentioned above.  


CIQ CVE Remediation - Frequently Asked Questions


## Introduction

This article provides a step-by-step guide to manually disable the screensaver lock screen setting on the GNOME desktop environment. Follow the instructions below to complete the process using `dconf` and `dconf-editor`.

## Problem

Users may need to disable the screensaver lock screen setting on their GNOME desktop environment for various reasons, such as preventing interruption during tasks.

## Prerequisites

Ensure that the `dconf` and `dconf-editor` packages are installed on your system. If they are not installed, use the following command to install them:

```bash
sudo dnf install -y dconf dconf-editor
```

## Steps to disable the screensaver lock-screen setting

1. **Launch dconf-editor**

   Open the `dconf-editor` by running the following command in your terminal:

   ```bash
   dconf-editor
   ```

   This will open the `dconf` GUI.

2. **Navigate to the screensaver settings**

   In the `dconf-editor`, navigate to the screensaver settings by following this path:

   ```
   org > gnome > desktop > screensaver
   ```

3. **Edit lock screen setting**

   - Click on `lock-enabled`. This will open a screen where you can change the value.
   - Toggle the `Use default value` to off.
   - Change the `Custom value` to `false`.
   - Click the blue check-mark at the bottom to confirm your changes.

   ![dconf-editor screensaver settings](public/images/articles/)
   
4. **Handling errors**

   If you encounter the following error in your terminal while attempting to save the settings:

   ```plaintext
   (dconf-editor:17050): dconf-WARNING **: [time]: failed to commit changes to dconf: Failed to execute child process "dbus-launch" (No such file or directory)
   ```

   This indicates that the `dbus-launch` utility is missing. Resolve this by exiting the GUI and installing the `dbus-x11` package:

   ```bash
   sudo dnf install -y dbus-x11
   ```

   After installing, relaunch the GUI and attempt to save your changes again. A system reboot may be required if the error persists.

## Validate the changes

To verify that the lock screen has been disabled, run one of the following commands:

```bash
gsettings get org.gnome.desktop.screensaver lock-enabled
```

or

```bash
dconf read /org/gnome/desktop/screensaver/lock-enabled
```

These commands should return `false`, indicating that the lock screen has been disabled.

## Conclusion

By following these steps, you should be able to manually disable the screensaver lock screen setting on your GNOME desktop environment. If you encounter any issues, ensure that all required packages are installed and correctly configured.


Disabling the Screensaver Lock Screen Setting


## Introduction

As specified in the [Warewulf Disk Management documentation](https://warewulf.org/docs/v4.5.x/contents/disks.html#rocky-linux), packages for [ignition](https://coreos.github.io/ignition/) are not available for the Rocky Linux 8.x series. They are available for the Rocky Linux 9.x series as part of the `Appstream` repository.

## Symptoms 

When you try to install `gdisk` or `ignition` on a Rocky Linux 8.x system, you will find it is not possible, due to the packages being unavailable in the repositories:

```bash
[root@Rocky-Linux-8-10-Test-Machine ~]# dnf install ignition
Last metadata expiration check: 0:04:50 ago on Fri 27 Dec 2024 05:46:44 AM UTC.
No match for argument: ignition
Error: Unable to find a match: ignition
```

## Resolution

### Prerequisites

You will require access to the `root` account or a user with escalated privileges.

### Build Ignition from source

When creating your container, add the following lines to your configuration:

```bash
&& (cd /tmp && git clone https://github.com/coreos/ignition.git) \
&& (cd /tmp/ignition && make all && make install) \
```

### Example container creation with Warewulf and ignition on Rocky Linux 8.10

Current disk structure of the nodes:

```bash
[root@Rocky-Linux-8-10-Warewulf-Controller-Node ~]# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sr0     11:0    1  1024M  0 rom  
vda    253:0    0   180G  0 disk 
├─vda1 253:1    0   260M  0 part /boot/efi
└─vda2 253:2    0 179.8G  0 part /
```

The Warewulf Rocky Linux 8.10 container was used for testing:

```bash
wwctl container import docker://ghcr.io/warewulf/warewulf-rockylinux:8 rockylinux-8 --build
```

**The following commands were all ran from the Controller Node:**

Created a 1GB `boot` partition for the Compute Node with `ext4` and wiped the filesystem at boot:

```bash
wwctl node set --diskname /dev/vda --diskwipe --partname boot --partsize=1024 --partnumber 1 --partcreate --fsname boot --fsformat ext4 --fspath /boot --fswipe
```

Assigned the rest of the disk space to the root filesystem, formatted with `btrfs`, and wiped the disk at boot for the Compute Node:

```bash
wwctl node set --diskname /dev/vda --partname root --partnumber 2 --partcreate --fsname root --fsformat btrfs --fspath / --fswipe
```

Output of how the Compute Node was configured:

```bash
[root@Rocky-Linux-8-10-Warewulf-Controller-Node ~]# wwctl node list -a warewulf-compute-node-1
  NODE                           FIELD                                                    PROFILE  VALUE                                                  
  warewulf-compute-node-1  Id                                                       --       warewulf-compute-node-1                          
  warewulf-compute-node-1  Comment                                                  default  This profile is automatically included for each node   
  warewulf-compute-node-1  ContainerName                                            default  rockylinux-8                                           
  warewulf-compute-node-1  Ipxe                                                     --       (default)                                              
  warewulf-compute-node-1  RuntimeOverlay                                           --       (generic)                                              
  warewulf-compute-node-1  SystemOverlay                                            --       (wwinit)                                               
  warewulf-compute-node-1  Root                                                     --       (initramfs)                                            
  warewulf-compute-node-1  Discoverable                                             --       false                                                  
  warewulf-compute-node-1  Init                                                     --       (/sbin/init)                                           
  warewulf-compute-node-1  Kernel.Args                                              --       (quiet crashkernel=no vga=791 net.naming-scheme=v238)  
  warewulf-compute-node-1  Profiles                                                 --       default                                                
  warewulf-compute-node-1  PrimaryNetDev                                            --       (default)                                              
  warewulf-compute-node-1  NetDevs[default].Type                                    --       (ethernet)                                             
  warewulf-compute-node-1  NetDevs[default].OnBoot                                  --       (true)                                                 
  warewulf-compute-node-1  NetDevs[default].Hwaddr                                  --       5a:00:05:3a:6f:5a                                      
  warewulf-compute-node-1  NetDevs[default].Ipaddr                                  --       10.25.96.4                                             
  warewulf-compute-node-1  NetDevs[default].Netmask                                 default  255.255.240.0                                          
  warewulf-compute-node-1  NetDevs[default].Gateway                                 default  10.25.96.3                                             
  warewulf-compute-node-1  NetDevs[default].Primary                                 --       (true)                                                 
  warewulf-compute-node-1  Disks[/dev/vda].WipeTable                                --       true                                                   
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[boot].Number                  --       1                                                      
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[boot].SizeMiB                 --       1024                                                   
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[boot].ShouldExist             --       true                                                   
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[root].Number                  --       3                                                      
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[root].ShouldExist             --       true                                                   
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/boot].Format          --       ext4                                                   
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/boot].Path            --       /boot                                                  
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/boot].WipeFileSystem  --       true                                                   
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/root].Format          --       btrfs                                                  
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/root].Path            --       /                                                      
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/root].WipeFileSystem  --       true
```

Ran `exec` to access the Rocky Linux 8.10 container:

```bash
wwctl container exec rockylinux-8 /bin/bash
```

Set up the environment, then cloned and built the `Ignition Partition Management Module` from inside the container:

```bash
dnf groupinstall -y "Development Tools"
dnf install -y gdisk git libblkid-devel make 
wget https://go.dev/dl/go1.23.4.linux-amd64.tar.gz
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.23.4.linux-amd64.tar.gz
```

* Added the following to the end of the ~/.profile file:

```bash
cat << "EOF" | tee ~/.profile
export PATH=$PATH:/usr/local/go/bin
EOF
```

* Added `~/.profile` to `$PATH`:

```bash
source ~/.profile
```

* Installed `ignition`:

```bash
cd /tmp && git clone https://github.com/coreos/ignition.git
cd /tmp/ignition && make all && make install
```

Ran `exit` to then leave and rebuild the container.

Deployed the container to the Compute Node by restarting the node.

## Notes

The above method is not officially supported by CIQ and is considered a workaround until the `gdisk` and `ignition` packages are available in the Rocky Linux 8.x repositories.

## References & related articles

Ignition Filesystem Documentation: https://coreos.github.io/ignition/operator-notes/#filesystem-reuse-semantics/

Warewulf Disk Management: https://warewulf.org/docs/v4.5.x/contents/disks.html#disk-management/


How to Setup Ignition with Warewulf on Rocky Linux 8.x

  
## Introduction

CIQ had a customer issue, where the customer was attempting to set up local users and then authenticate using `Kerberos` to their `Active Directory` server. The following guide will explain how this was resolved.

## Problem

The customer was unable to authenticate via `ssh` to their Rocky Linux 9.2 LTS node using a local user account stored on the node. No passwords were locally saved in the `/etc/shadow` file and the passwords were only available on the `Active Directory` server.

The customer's configuration of `sssd`, `Kerberos` and `Active Directory` had worked without issue on CentOS 7.9, CentOS 8.1 and Rocky Linux 8.5. It was only on Rocky Linux 9.2 LTS where the customer was facing issues.

## Symptoms

When attempting to log in to the server via `ssh`, the following authentication messages were observed under `/var/log/secure`:

```bash
Dec 9 11:07:31 <server_name> sshd[1155]: Received signal 15; terminating.
Dec 9 11:07:31 <server_name> sshd[79752]: Server listening on 0.0.0.0 port 22.
Dec 9 11:07:31 <server_name> sshd[79752]: Server listening on :: port 22.
Dec 9 11:10:04 <server_name> unix_chkpwd[79934]: check pass; user unknown
Dec 9 11:10:04 <server_name> unix_chkpwd[79934]: password check failed for user (<username>)
Dec 9 11:10:04 <server_name> sshd[79919]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<ip_address> user=<username>
Dec 9 11:10:44 <server_name> sshd[79919]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<ip_address> user=<username>
Dec 9 11:10:44 <server_name> sshd[79919]: pam_sss(sshd:auth): received for user <username>: 4 (System error)
```

Account retrieval errors could also be seen via the `/var/log/sssd/sssd_pam.log` and `/var/log/sssd/sssd_nss.log` files:

```bash
[root@<server_name> ~]# cat /var/log/sssd/sssd_pam.log
(2024-12-16 13:45:58): [pam] [server_setup] (0x3f7c0): Starting with debug level = 0x0070
(2024-12-16 13:47:20): [pam] [cache_req_common_process_dp_reply] (0x3f7c0): [CID#1] CR #3: Could not get account info [<account_number>]: SSSD is offline

[root@<server_name> ~]# cat /var/log/sssd/sssd_nss.log
(2024-12-16 13:45:58): [nss] [server_setup] (0x3f7c0): Starting with debug level = 0x0070
(2024-12-16 13:47:20): [nss] [cache_req_common_process_dp_reply] (0x3f7c0): [CID#5] CR #17: Could not get account info [<account_number>]: SSSD is offline
```

## Resolution

The customer was utilizing `id_provider = files` in their `/etc/sssd/sssd.conf` file. Since the Rocky Linux 9.x series, [this is a deprecated option in sssd](https://sssd.io/docs/files-provider-deprecation.html).

Instead, they added the following settings into their `sssd.conf` file:

```bash
id_provider = proxy
krb5_server = example.com:88
proxy_lib_name = files
```

Then restarted the `sssd` service:

```bash
systemctl restart sssd`
```

The customer's `sssd.conf` file ultimately looked like the following (all sensitive customer information has been obfuscated):

```bash
[sssd]
    services = nss, pam
    domains = EXAMPLE.COM
    default_domain_suffix = example.com

[domain/EXAMPLE.COM]
    id_provider = proxy
    proxy_lib_name = files
    auth_provider = krb5
    krb5_realm = EXAMPLE.COM
    krb5_server = example.com:88
    krb5_validate = false
    krb5_ccachedir = /var/tmp
    krb5_keytab = /etc/krb5.keytab

[nss]
filter_groups = root,<local user>
filter_users = root,<local user>
```

In addition, the customer's `/etc/kr5b.conf` file:

```bash
includedir /etc/krb5.conf.d/

[logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

[libdefaults]
    default_realm=EXAMPLE.COM
    dns_lookup_realm = true
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    default_ccache_name = KEYRING:persistent:%{uid}

[realms]
     EXAMPLE.COM = {
     kdc = example.com:88
     admin_server = example.com:749
     }
     example.com = {
     kdc = example.com:88
     admin_server = example.com:749
     }

[domain_realm]
      example.com = EXAMPLE.COM
      example.com = EXAMPLE.COM

[appdefaults]
 pam = {
   debug = true
   ticket_lifetime = 36000
   renew_lifetime = 36000
   forwardable = true
   krb4_convert = false
 }
```

Furthermore, the customer's `/etc/nsswitch.conf` file:

```bash
# In order of likelihood of use to accelerate lookup.
shadow:     files
hosts:      files dns myhostname

aliases:    files
ethers:     files
gshadow:    files
# Allow initgroups to default to the setting for group.
# initgroups: files
networks:   files dns
protocols:  files
publickey:  files
rpc:        files
```

## Troubleshooting notes

Adding `debug_level = 9 ` to each section of the `sssd.conf` file, yields more verbose logs under `/var/log/sssd/*`

When troubleshooting the issue, the following logs are essential:

```bash
/var/log/sssd/*
/var/log/secure
```

In addition, these files are also important to check:

```bash
/etc/kr5b.conf
/etc/nsswitch.conf
/etc/pam.d/password-auth
/etc/pam.d/system-auth
/etc/sssd/sssd.conf
```

Next are these command outputs to run:

```bash
getent passwd <username>
klist
kinit <username>
ssh -o PubkeyAuthentication=no -vvv -k <username>@<server_name>
```

Finally, generating an `sosreport` from the node.

## References & related articles

[Mapping local users to Kerberos principals with SSSD](https://blog.oddbit.com/post/2015-07-16-mapping-local-users-to-kerbero/)

[sssd.conf man page](https://manpages.org/sssdconf/5)

[sssd-files man page](https://man.archlinux.org/man/sssd-files.5.en)

[sssd-krb5 man page](https://linux.die.net/man/5/sssd-krb5)


Enabling Local User Login with Active Directory Authentication using Kerberos


## Introduction

When installing Rocky Linux in a cloud environment, all directories, including `/tmp`, will typically be installed on a single volume. If you try to use a separate partition or volume for `/tmp`, you might encounter an issue where the entire root filesystem becomes read-only after a reboot. This occurs because the `systemd` `tmp.mount` unit is sometimes masked by default.

When a `systemd` unit is masked, it is linked to `/dev/null`, making it impossible to start. This differs from disabling a service, which prevents it from running at startup, but does allow it to be manually started or started by another process or unit.

You want the `tmp.mount` unit to be disabled so it does not start by default. However, you need `systemd` to start the `tmp.mount` service when it runs the `systemd-fstab-generator` at boot, which converts `/etc/fstab` entries into native `systemd` units. If `tmp.mount` is masked, this process fails, causing the system to mount the root directory as read-only.

## Symptoms

After you add an entry for `/tmp` in `/etc/fstab` and then reboot, you may see many errors related to a read-only filesystem. You will likely also see the following error when attempting to create a file:

```bash
touch /var/tmp/test
touch: cannot touch 'var/tmp/test': Read-only file system
```

## Resolution

Telling `systemd` to unmask this unit will allow you to mount `/tmp` according to how `/etc/fstab` is configured:

```bash
sudo systemctl unmask tmp.mount
```

If you get a read-only filesystem error when attempting this, you need to remount the root volume first:

```bash
sudo mount -o remount,rw /
```

Once the filesystem is remounted and you have verified you can write to it, rerun the `unmask` command. After a reboot, your server should start up as expected.

There is no need to enable this unit as `systemd` will dynamically generate it and start it at boot.

## Root Cause

This issue arises because cloud servers are primarily used for network-specific tasks such as web hosting. 

As a result, the need for an in-memory `/tmp` directory (which is generated by default when you start `tmp.mount`), is largely negated because the network is significantly slower than the disk that `/tmp` would run on. 

Additionally, since memory is a significant factor in the cost of cloud services, Rocky Linux does not rely on memory-based `tmpfs` directories. Moreover, the need for a separate partition or volume for `/tmp` is only necessary in minimal use cases and is unnecessary for the majority of workloads.

## References & related articles

freedesktop.org's page on `systemd-fstab-generator`: [Link](https://www.freedesktop.org/software/systemd/man/latest/systemd-fstab-generator.html)
Rocky Linux Admin Guide [Link](https://rocky-linux.github.io/documentation/RockyLinuxAdminGuide.pdf)

How to Avoid a Read-only Filesystem When Using a Separate Partition or Volume for /tmp


## Introduction

The key to this problem is *entropy*. The FIPS openssl version is built with strict requirements for entropy to provide the necessary security guarantees: it fills up "pools" of truly random numbers when it starts up. If these pools aren't filled, then applications that call the openssl library will block until it can gather enough random material to proceed. Regular openssl doesn't do this: if it can't get enough true randomness, it uses fuzzy pseudorandomness and continues. It is important to understand that FIPS is not meant to be run in pieces. It is a holistic system. The kernel provides the entropy to the openssl library via a system call.

The RHEL/Rocky 9 kernel (and upstream 5.10, 5.15, etc. kernels) seem to provide enough entropy more or less built-in. CentOS 7 (kernel 3.10.0) and RHEL/Rocky 8 (kernel 4.18) do not in some cases.

## Problem

When using openssl fips package on a CentOS/Rocky 8 system you may see the following timeouts when using dnf with the FIPS openssl libraries installed:

```
dnf list
Rocky Linux BaseOS         0.0  B/s |   0  B     17:45
Errors during downloading metadata for repository 'baseos':
 - Curl error (28): Timeout was reached for https://.../BaseOS/x86_64/os/repodata/repomd.xml [SSL connection timeout]
 - Curl error (28): Timeout was reached for https://.../BaseOS/x86_64/os/repodata/repomd.xml [Operation timed out after 99165 milliseconds with 0 out of 0 bytes received]
Error: Failed to download metadata for repo 'baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
```

Also, sshd may freeze on startup, preventing remote access to a system.

## Resolution

Fortunately, the solution is simple: On any CentOS 7/Rocky 8 host, if you want to run containers with the FIPS openssl library installed, you need to do this (on the host, not inside the Rocky 8 container):

```
yum install rng-tools

systemctl enable --now rngd
```

The rngd daemon feeds extra, hardware generated, high-quality sources of entropy into the kernel, thus preventing the entropy pool from becoming exhausted.


Configuring Entropy on CentOS7/RockyLinux8 for FIPS OpenSSL


## Introduction

Fuzzball Orchestrate is architected as a collection of microservices deployed in a Kubernetes (K8s)
cluster. To understand what is happening with the Fuzzball deployment, you must be able to inspect
the status of the pods supporting these containerized microservices, and gather relevant logs.

## Checking pod status

Any Fuzzball troubleshooting starts with confirming the presence and inspecting the status of
containers/microservices within their respective pods.

You can view and inspect the status of pods hosting containerized microservices using the standard
K8s control plane tool, `kubectl`. After executing the following `kubectl` command, a table
is displayed giving information about all of the pods within the K8s cluster.

```text
# # kubectl get pods -A
NAMESPACE            NAME                                                       READY   STATUS      RESTARTS      AGE
cert-manager         cert-manager-56f459fcb7-j7wmt                              1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-56f459fcb7-m8gq2                              1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-56f459fcb7-qj9zd                              1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-cainjector-78f497d9d8-5fc5r                   1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-webhook-5bf5cccd59-qg5qs                      1/1     Running     1 (35m ago)   19d
cert-manager         trust-manager-778f7b488-gdmrh                              1/1     Running     1 (35m ago)   19d
database             database-debug-0                                           1/1     Running     1 (35m ago)   19d
database             database-postgresql-0                                      1/1     Running     1 (35m ago)   19d
fuzzball-system      fuzzball-operator-controller-manager-55cc86985f-c7tzn      2/2     Running     2 (35m ago)   19d
fuzzball             audit-archive-28836720-4wcf5                               0/1     Completed   0             2d6h
fuzzball             audit-archive-28838160-86zjk                               0/1     Completed   0             30h
fuzzball             audit-archive-28839600-tbhj2                               0/1     Completed   0             6h25m
fuzzball             fuzzball-account-7749678b7c-m272g                          1/1     Running     1 (35m ago)   19d
fuzzball             fuzzball-account-external-5f5cd6f56-86k4z                  1/1     Running     1 (35m ago)   19d
fuzzball             fuzzball-account-worker-7f69c98747-gbtvc                   1/1     Running     4 (33m ago)   19d
fuzzball             fuzzball-admin-0                                           1/1     Running     1 (35m ago)   19d
[...snip]
```

If you want to view only pods running within the `fuzzball` namespace, you can change the command
like so:

```text
# kubectl get pods -n fuzzball 
NAME                                             READY   STATUS      RESTARTS       AGE
audit-archive-28836720-4wcf5                     0/1     Completed   0              2d8h
audit-archive-28838160-86zjk                     0/1     Completed   0              32h
audit-archive-28839600-tbhj2                     0/1     Completed   0              8h
fuzzball-account-77bc99544-n2dgb                 1/1     Running     0              119s
fuzzball-account-external-7977f8c5dc-hmfhf       1/1     Running     0              119s
fuzzball-account-worker-6b646c8b76-jq2sf         1/1     Running     0              119s
fuzzball-admin-0                                 1/1     Running     1 (173m ago)   19d
fuzzball-audit-74bb656b6b-lltx4                  1/1     Running     0              119s
fuzzball-auth-spicedb-57bb79cf9b-k2zh7           1/1     Running     0              119s
fuzzball-dns-569f675947-cf59z                    1/1     Running     0              119s
fuzzball-kube-68fc8d8b8d-f7gkg                   1/1     Running     0              119s
fuzzball-log-845fc88dd9-bvr4c                    1/1     Running     0              119s
fuzzball-openapi-686488649-7fj2s                 1/1     Running     0              119s
fuzzball-organization-7f7f5f6878-qs55p           1/1     Running     0              119s
fuzzball-organization-external-cc78884d5-z56j5   1/1     Running     0              118s
fuzzball-organization-worker-55599cd8d8-htknd    1/1     Running     0              118s
fuzzball-permission-external-55d965c957-n5fsv    1/1     Running     0              118s
fuzzball-schedule-0                              1/1     Running     0              170m
[...snip]
```

Note the values in the `READY` column indicating `1/1`, (or potentially `2/2`, etc.) and the
`STATUS` column indicating `Running`. Any pods that do not report a `Running` or `Completed` status,
or those that report fewer instances than expected should be investigated further by inspecting
logs.

## Retrieving and Inspecting logs

While provisioning Fuzzball within a K8s cluster (using `kubectl apply -f fuzzball.yaml`), you can
watch the log output with the following command:

```text
# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1
```

You can stream logs from an individual pod using the following syntax. Note that the namespace must
be specified since Fuzzball pods are not deployed in the "default" namespace. For example, to view
the logs streaming from the `fuzzball-schedule-0` pod, you could use the following command. (Omit
the `--follow --tail=0` options/arguments if you want to view the log at a single instant without
streaming it.)

```text
# kubectl logs -n fuzzball fuzzball-admin-0 --follow --tail=0
```

The `kubectl logs` command outputs verbose logs in json format. These can be filtered and
parsed into human-readable output using `jq` like so:

```text
# kubectl logs -n fuzzball fuzzball-admin-0 | \
    jq -r 'select(.level != "debug") | [.ts, .msg, .Error] | @tsv'
```

The `kubectl` utility does not provide a straightforward command for capturing all Fuzzball logs in
a single stream. However, the logs for a collected Fuzzball “app” can be inspected using a label
selector like so. (This is the strategy used above to stream logs during the provisioning process.)

```text
# kubectl logs --prefix --namespace=fuzzball --all-containers \
    --max-log-requests=99 --tail=0 --follow \
    --selector='app.kubernetes.io/name=fuzzball-schedule'
```

The list of available labels can be viewed using `kubectl get pods -n fuzzball --show-labels`.

Finally, it may be useful to gather a full log dump (for example, to send to CIQ support). This can
be accomplished by requesting logs from all app labels. The app labels may change with different
versions of Fuzzball and can be inspected as described above.  Here is an example:

```text
# kubectl logs --prefix --all-containers --max-log-requests=99 \
    --namespace=fuzzball \
    --selector='app.kubernetes.io/name in (fuzzball-account, fuzzball-admin, fuzzball-audit, fuzzball-auth, fuzzball-dns, fuzzball-kube, fuzzball-log, fuzzball-openapi, fuzzball-organization, fuzzball-permission, fuzzball-schedule, fuzzball-secret, fuzzball-storage, fuzzball-sync, fuzzball-ui, fuzzball-user, fuzzball-workflow)'
```

## Restarting Fuzzball

You can invoke the command `kubectl rollout restart` to restart the Fuzzball deployment. For
example:

```text
# kubectl rollout restart deployment -n fuzzball 
```

Other namespaces that may be pertinent for restart include

- `database` - provides the PostgreSQL database that backs Fuzzball Orchestrate state
- `ingress` - maps HTTP(S) URLs to relevant Fuzzball services, including the Fuzzball UI, Fuzzball API, and Keycloak
- `kyverno` - provides a policy engine used by Fuzzball Orchestrate
- `workflow` - provides the temporal service that is used by Fuzzball Orchestrate
- `identity` - provides a Keycloak instance for identity and access management

## Uninstalling Fuzzball

If a deployment failed or you have a need to uninstall a deployed Fuzzball installation, you may
need to tear down the installation and start fresh. The following command will delete an existing
Fuzzball deployment, allowing you to start over if needed.

```text
# kubectl delete FuzzballOrchestrate fuzzball-orchestrate
```

You can watch the deprovisioning process as it occurs with the following command:

```text
# kubectl logs -n fuzzball-system -l app.kubernetes.io/name=fuzzball-operator -f --tail=-1
```

You may also want to remove the `substrate` configuration directory that is shared out to compute
nodes via NFS.  For instance:

```text
# # rm -rf /srv/fuzzball/shared/substrate
```

This directory will be recreated if you re-provision your Fuzzball instance.


Basic Troubleshooting Procedures


## Introduction

The need for secure and efficient file transfer solutions has become paramount. Whether it is sharing sensitive data, collaborating on projects, or automating workflows, a reliable Secure File Transfer Protocol (SFTP) server can streamline these processes while ensuring data integrity and confidentiality.

The CIQ Secure File Exchange (SFE) service is an SFTP, FTP/S, HTTP/S, and WebDAV server. It provides a flexible foundation for managing file transfers while offering an efficient and scalable solution for storing and retrieving data. The service supports private virtual folders per user, which can also be shared with other users as well.

## How to access the service

To have an SFE account generated, please create a ticket via the CIQ Support Portal. A member of support will then generate your account and securely send you the login credentials via the ticket.

Login to the [CIQ SFE Client Portal](https://sftp.ciq.com/web/client/login) using your provided credentials.

You will then be prompted to change your password. Please do so and store the new password in a secure location.

## How to upload files

Files can be uploaded in two ways: via your web browser or the CLI.

### Web browser method

Log in with your credentials at the [CIQ SFE Client Portal](https://sftp.ciq.com/web/client/login).

Navigate to the `My Files` section and press the `Upload files` button.

![An image showing the CIQ SFE Client's My Files section and highlighting the "Upload" button that can be clicked to open the file upload dialogue box.](/images/articles/ciq_sfe_file_upload_1.webp)

Select your files and then hit `Submit`:

![An image from the CIQ SFE Client, highlighting the area where the user can upload their files and then clicking the "Submit" button to start the upload.](/images/articles/ciq_sfe_file_upload_2.webp)

The file will be uploaded to your virtual folder. Please inform the Support Team via the relevant support ticket and they can retrieve the files and start their investigation.

**If you would like to further customize how you share your files, please follow the steps below:**

Under `My Files`, click the checkboxes next to the files you uploaded and then click the `Share` button:

![An image from the My Files section of the CIQ SFE Client, with the files that the user wants to share having their checkboxes ticked.](/images/articles/ciq_sfe_file_upload_4.webp)

![An image from the My Files part of the CIQ SFE Client, highlighting the Share button to click, so that the user can share their files.](/images/articles/ciq_sfe_file_upload_3.webp)

At the next screen, add a `Name` for your share, set a `Password` and `Expiration` date if you desire and then select `Submit`:

![An image of the "Add a new share" section of the CIQ SFE Client, showing a "Name" can be added for the share, the "Scope" can be changed from "Read" to "Write" to "Read and Write", a Password can be added, as well an Expiration date. Finally the "Submit" button is highlighted in green for the user to click and the share can be created.](/images/articles/ciq_sfe_file_upload_5.webp)

Then in the `Shares` screen, select the share that you created and click the `Link` button at the top of the page:

![An image showing the test share that was created in the previous step and has been selected for sharing by the user.](/images/articles/ciq_sfe_file_upload_7.webp)

![An image displaying the "Shares" section of the CIQ SFE Client, showing the "Link" button where links to the share can be generated.](/images/articles/ciq_sfe_file_upload_6.webp)

Finally, copy the link type you would like to share and include the link in the relevant ticket. The Support Team can then easily access your files:

![An image showing that once the "Link" button is clicked, three links are generated: 1) The user can download the share as a zip file. 2) The user can access a directory to view all of the files. 3) If only a single file is shared, the user can download the file uncompressed.](/images/articles/ciq_sfe_file_upload_8.webp)

### CLI method

Run the following command to access the CIQ SFE service via the CLI:

```bash
sftp -P 2022 <YOUR_USERNAME>@ciq.com@sftp.ciq.com
```

Enter your password when prompted.

Create a directory where you would like to store your files:

```bash
sftp> mkdir <YOUR_DIRECTORY>
```

Upload your files using the `put` command:

```bash
put ./text.txt ./test/
```

An example is below of the author uploading a file called `test.txt` to a directory in the CIQ SFE service called `test`:

```bash
sftp> put ./test.txt test/
Uploading ./test.txt to /test/test.txt
test.txt                                                                                                                                      100%   13     0.1KB/s   00:00    
sftp> 
```

Once uploaded, please check that the files were uploaded successfully. An example is shown below:

```bash
sftp> ls -l test/ 
-rwxr-xr-x    1 991      991            13 Jan 20 02:32 test.txt
```

Please inform the Support Team via your ticket that the files have been uploaded successfully.

## Important!

For all methods listed above, **please provide a sha256sum in the ticket for each file that is in the multiple gigabytes**. This allows the Support Team to easily verify if an upload has been successful or not and can continue their investigation without further interruption.

If you run into any problems whilst uploading files to the CIQ SFE service, please don't hesitate to make the Support Team aware and they will assist you with the uploading process.

How to Upload Files to the CIQ Secure File Exchange (SFE) Service


## Introduction

It is inevitable that hardware will fail and there are hundreds of possible scenarios.

This guide will detail troubleshooting steps which can be used to identify hardware errors and how to rectify them.

## Problem

A customer informed CIQ that a node in their production cluster had spontaneously crashed and then rebooted itself.

## Symptoms

Checking `/var/log/messages` from the `sosreport`, there were hundreds of `[Hardware Error]: Hardware error from APEI Generic Hardware Error Source` messages observed.

## Resolution

### Prerequisites

Access to the `root` user or a user with escalated privileges.

### Creating an sosreport

When creating a ticket with CIQ, generating and attaching an `sosreport` to the ticket is very helpful. This allows the support team to start their investigation, without further having to ask the customer to generate one.

Install the `sos` package:

```bash
dnf install -y sos
```

Run the `sos report --batch` command to perform unattended data gather of your system.

Once complete, you will find the `sosreport` under `/var/tmp/sosreport-hostname-yyyy-mm-dd-abcdefg.tar.xz`. An example is below:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# ls -l /var/tmp/ | grep sosreport
-rw-------. 1 root root 11094320 Dec 27 02:33 sosreport-Rocky-Linux-9-5-Test-Machine-2024-12-27-elpmjjq.tar.xz
-rw-r--r--. 1 root root       65 Dec 27 02:33 sosreport-Rocky-Linux-9-5-Test-Machine-2024-12-27-elpmjjq.tar.xz.sha256
```

Please upload the `sosreport` to the ticket.

### Analysis of the issue

When observing behavior such as crashes or random reboots, the best log source is `/var/log/messages` - this records the global messages that are generated from the various services running on your server.

Hardware errors are usually identified by the kernel in the `/var/log/messages` log with a `[Hardware Error]` prefix. These messages come from the ACPI Platform Error Interface.

Using a combination of tools such as `grep`, `awk`, and `uniq` you can quickly find and triage repeating errors. The following in `/var/log/messages` from a customer's `sosreport` shows hundreds of memory-related errors:

```bash
grep "Hardware Error" ./sosreport-<NODE_NAME>-2024-12-24-gwwbeqs/var/log/messages | awk '{print $8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20}' | grep "memory error" | uniq -c
 710 section_type: memory error
```

In addition to when a `Hardware Error` occurs, you will also encounter messages of `Corrected error, no action required`. While the error may be temporarily corrected, if these messages continue, it is a sign of a more serious fault. In that scenario, it is best to contact the hardware vendor and replace the affected part outright.

### Recommendations

In the above memory fault example, the recommendation to the customer was to change out their memory modules and they then contacted their hardware team for further analysis.

## References & related articles

`awk` man page: https://man7.org/linux/man-pages/man1/awk.1p.html

`grep` man page: https://man7.org/linux/man-pages/man1/grep.1.html

`sosreport` man page: https://linux.die.net/man/1/sosreport

`uniq` man page: https://man7.org/linux/man-pages/man1/uniq.1.html


How to Identify Hardware Faults in Rocky Linux


## Introduction
A quick intro to the topic at hand.

In some technical processes, reading from and writing to registers may lead to inconsistencies or invalid values due to the simultaneous execution of read and write operations. This article provides an overview of this issue and offers potential solutions for troubleshooting it. 

## Problem
If gathersing the pci information from /sys (sysfs) the data shows pci link speed and status. In some cases, the LnkSta that is returned indicates the following "16GT/s(downgraded)  x4(downgraded)". While 16GT/s and x4 might be the correct speed and width, the ouput shows "downgraded".

```
$ lspci | grep downgr
...
        LnkSta: Speed 8GT/s (downgraded), Width x8 (ok)
        LnkSta: Speed 16GT/s (ok), Width x8 (downgraded)
...
```
### This could match the following criteria

- Inconsistencies in data when comparing multiple reads from the same register
- Observing incorrect or unexpected values that do not align with the expected behavior.
- Frequent occurrences of invalid values (e.g., once or twice a week).

## Resolution

While this is not a resolution to the problem, this could help gathering information that you can bring to your hardware provider.

Below is the lspci code that emits the string: “(downgraded)”. Just so we know what criteria is being used here:

```
static char *link_compare(int type, int sta, int cap)
{
  if (sta > cap)
    return " (overdriven)";
  if (sta == cap)
    return "";
  if ((type == PCI_EXP_TYPE_ROOT_PORT) || (type == PCI_EXP_TYPE_DOWNSTREAM) ||
      (type == PCI_EXP_TYPE_PCIE_BRIDGE))
    return "";
  return " (downgraded)";
}
```
This is called from: cap_express_link()

```
  w = get_conf_word(d, where + PCI_EXP_LNKSTA);
  sta_speed = w & PCI_EXP_LNKSTA_SPEED;
  sta_width = (w & PCI_EXP_LNKSTA_WIDTH) >> 4;
  printf("\t\tLnkSta:\tSpeed %s%s, Width x%d%s\n",
        link_speed(sta_speed),
        link_compare(type, sta_speed, cap_speed),
        sta_width,
        link_compare(type, sta_width, cap_width));
```
The link_compare() code came from [this discussion:](https://lore.kernel.org/all/20210318171122.GA151712@bjorn-Precision-5520/T/)

In sum, it’s checking the status speed from `PCI_EXP_LNKSTA_SPEED` and if it’s less than cap_speed then it’s always reporting “(downgraded)” for devices other than `PCI_EXP_TYPE_ROOT_PORT`, `PCI_EXP_TYPE_DOWNSTREAM`, `PCI_EXP_TYPE_PCIE_BRIDGE`.

The PCI configuration data, including speed and width information that lspci code reads and potentially reports as “downgraded,” is pulled from the [sysfs](https://docs.kernel.org/PCI/sysfs-pci.html) entries under `/sys/bus/pci/devices`. As an example, for NVMe devices, the data is accessed from the corresponding entries for each NVMe device under `/sys/bus/pci/devices/[DEVICE_ID]/config`.
 
The reading of PCI configuration data is managed by the kernel's PCI subsystem, specifically within the PCI sysfs interface and related functions. The primary functions involved include:
```
pci_read_config() in drivers/pci/pci-sysfs.c
pci_user_read_config_##size in drivers/pci/access.c
```
These functions handle the reading of configuration data directly from the hardware registers of the PCI devices. 

If you use a script that reads these data, is good to understad that it is just getting raw data out of the device on the PCI bus.
 
For debugging you could try using dmesg and other system logs to check for any hardware events or errors reported by the kernel that might indicate link renegotiation or other PCIe issues. 

Look for AER (Advanced Error Reporting) messages which can sometimes provide clues about PCIe link issues:
```
$ dmesg | grep AER
```

## Possible Causes

1. Simultaneous read and write operations on the same register: This can result in capturing an intermediate state instead of the final value, leading to inconsistencies.
2. Incorrect timing between read and write operations: If there's not enough time between reading and writing, it may lead to the capture of an incorrect or invalid value.
3. Hardware malfunctions: Faulty hardware components or software glitches can cause unexpected behavior in register values.


## References & related articles 
[lspci: Don't report PCIe link downgrades for downstream ports](https://lore.kernel.org/all/20210318171122.GA151712@bjorn-Precision-5520/T/)
[Accessing PCI device resources through sysfs](https://docs.kernel.org/PCI/sysfs-pci.html)


Inconsistent Values When Reading and Writing PCI Registers


## Introduction
WareWulf includes the ability to enter into an image via a shell. This allows you to modify the container image and install components like Nvidia drivers easily and quickly.

## Resolution

We first need to start by either copying an existing container or downloading a new container. You can use `wwctl container list` to view your current images if you would like to build off an existing container. You can of course edit any existing container, however, we always recommend either workling off of a copy or creating a backup first. 
 
In this example we are going to start by downloading a fresh copy of Rocky Linux 9. We can do so by grabbing an image from the WareWulf repository. 

```
wwctl container import docker://ghcr.io/warewulf/warewulf-rockylinux:9 rockylinux-nvidia-9
```
 
The name of our container will be `rockylinux-nvidia-9` in this example. Feel free to change this to match your environment or to add additional information such as the date or version. Once the container is finished downloading, we can enter into this container's shell and install the Nvidia drivers.

```
wwctl container shell rockylinux-nvidia-9
```
 
You can read more about how to install the Nvidia drivers [here](https://docs.nvidia.com/cuda/pdf/CUDA_Installation_Guide_Linux.pdf). The following are example steps for Rocky Linux 9. 

```
# Install necessary packages and add the Nvidia repository
dnf -y install dnf-plugins-core epel-release kernel-headers
dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/$(arch)/cuda-rhel9.repo
# Install the latest Nvidia driver
dnf -y module install nvidia-driver:latest-dkms
```
 
We can verify this installed correctly by checking for the dynamically loaded kernel

```
# dkms status | grep nvidia
nvidia/555.42.02, 5.14.0-427.20.1.el9_4.x86_64, x86_64: installed
```

Once you have finished installing the driver and any other applications you need, type `exit` to leave the container. Warewulf should rebuild the container once you are finished but you can run this manually to verify the container was built.

```
wwctl container build
```
 
Running this command will not return anything if the image has already been built/updated. If applicable we can sync our local users into our container. You can read more about the `syncuser` subcommand [here](https://warewulf.org/docs/main/contents/containers.html#syncuser). 
 
```
wwctl container syncuser --write rockylinux-nvidia-9
```
 
Once the syncuser command completes, we can assign this new image to a node. Assigning it directly to a node allows us to boot and test our newly created container before pushing it out to a profile. You can skip directly to assigning this to a profile if you desire. 


```
wwctl node set node1 --container rockylinux-nvidia-9
```


Once you have tested your container image you can assign the container to the necessary profile. In our example we are sticking with the default profile

```
wwctl profile set default --container rockylinux-nvidia-9
```
 
And finally we can build the overlay

```
wwctl overlay build
```

Upon reboot, your nodes will now log in with the newly created container image! You can find more information about WareWulf containers as well as more advanced configuration examples and instructions in the [WareWulf documentation](https://warewulf.org/docs/main/contents/containers.html). 
 
## References & related articles
[WareWulf Documentation](https://warewulf.org/docs/main/contents/containers.html)  
[WareWulf Rocky Linux 9 Nvidia Container Example](https://github.com/warewulf/warewulf-node-images/blob/main/examples/rockylinux-9-nvidia/Containerfile)  
[Hands on Warewulf: Solving Cluster Provisioning & Management](https://www.youtube.com/watch?v=7w8dAU9RSpE)  
[Warewulf: Deep Dive, Use Cases, and Examples](https://www.youtube.com/watch?v=EpVDeesAq4c)  


Install Nvidia Drivers in a WareWulf Container


## Introduction
As mentioned [here](https://warewulf.org/docs/main/contents/ipmi.html):
It is possible to control the power or connect a console to your nodes being managed by Warewulf by connecting to the BMC through the use of IPMI

## Problem
The IPMI command works for all nodes after reboot, using their ipmitool command and node's IP address:
```
# ipmitool -I lanplus -H 10.0.0.11 -U root -P <PASSWORD> power status
Chassis Power is on
```

However, it does not work with wwctl power command:

```
wwctl power status cpu-20-10
ERROR  : cpu-20-10: Get Session Challenge command failed
Error: Unable to establish LAN session
Error: Unable to establish IPMI v1.5 / RMCP session
ERROR: exit status 1
```

## Resolution
You may need to configure IPMI to use **lanplus**. That could be achievable by setting `--ipmiinterface=lanplus`. Example:

```
wwctl profile set default \
> --netmask=255.255.255.0 --gateway=10.10.10.30 \
> --ipmiuser=root --ipmipass=[redacted] \
> --ipminetmask=255.255.255.0 --ipmiinterface=lanplus
```

## References & related articles
[Deploying a Dell PowerEdge and Cornelis Omni-Path Cluster with Warewulf](https://ciq.com/blog/deploying-a-dell-poweredge-and-cornelis-omni-path-cluster-with-warewulf/)

IPMI not working with wwctl power command for the LANPlus interface


## Introduction
If you are on an LTS version of Rocky Linux and would like to transition to the latest upstream version of Rocky Linux you need to revert the repositories and the release back to the default repositories

## Resolution
We can accomplish migrating from CIQ LTS for Rocky Linux 9.2 to Rocky Linux 9.4 running these commands:
 
```
sudo mtn dnf disable lts-9.2
sudo dnf swap ciq-lts92-rocky-release rocky-release --allowerasing --skip-broken
sudo dnf distrosync 
```

## Notes

- The dnf swap command will replace all of your Rocky repo's in /etc/yum/repos.d/ with the default repos. If you use any custom repo URLs, back them up before running this command.
- The dnf distrosync command will upgrade your server to 9.4. Please ensure you have the necessary backups before proceeding. 
- If you use secure boot, test this on a non production server first to ensure there are no issues.
- If dnf distrosync finishes but fails to install a package or two, you can run `dnf upgrade` to finish upgrading any missed packages.

## References & related articles
[CIQ Long Term Support](https://ciq.com/services/long-term-support/)

Migrate From Rocky Linux LTS to Rocky Linux


## Introduction

The Fuzzball Web GUI allows you to view logs from your running or completed job.

## Problem

Under some circumstances, a user may click on the "Logs" button within the job section of the
workflow dashboard and receive the message "No log output" even though the job completed
successfully and logs were expected. The problem may be that user does not have the appropriate
permissions to view the logs, or that the browser cannot access the API endpoint because Fuzzball
was installed using self-signed certificates.

## Symptoms

Here is an example showing the problem. The user has clicked on the "Logs" button for a workflow
that was expected to produce logs and is seeing the message "No log output".

![No log output in GUI](/images/articles/no-logs.png)

## Resolution

### Ensure you have permission to view logs

First, ensure that you have the appropriate permissions to view logs in the workflow.  If you are
viewing a job in a workflow submitted by another member of your shared account, you might need to
ask them for permission to view the logs.  They can add you as a job owner using the following CLI
command:

```text
$ fuzzball workflow add-owner <workflow UUID> <user UUID>
```

The workflow UUID and user UUID can be obtained with the `fuzzball workflow list` and `fuzzball
account list-members` commands respectively.  After you have been added as an owner you should be
able to view logs from jobs within the workflow.

### Self-signed certificate require exception

If you have the correct permission to view job logs, you should check to make sure that your browser
can access them from the API.  When self-signed certificates are used, the Fuzzball GUI asynchronous
access to the Fuzzball Orchestrate API can be blocked by your web browser's security policy, causing
an erroneous indication that no logs are available.

To correct this issue, you can direct your browser to the Fuzzball API endpoint.  The exact URI will
depend on the way that Fuzzball was installed, but it may be accessible by changing the string `ui`
in the Fuzzball URL to `api`.  For instance the following refers to the URL for the Fuzzball web GUI
and the API on an example installation respectively.

- https://ui.stable.fuzzball.ciq.dev/
- https://api.stable.fuzzball.ciq.dev/

In the case of a self-signed certificate, your browser may complain that the URL is unsafe and may
prompt you to create an exception.  After doing so, a browser refresh should allow you to view the
logs in the web GUI.


SSH Keys Issue on Rocky 9


## Introduction

A `mutex` (meaning `mutual exclusion`) prevents multiple threads from accessing the same resource simultaneously. When CIQ was running a `futex` test (a form of testing that checks the Linux kernel `futex` system call API), we observed `mutex` slowdown when running the `futex` testing on Rocky Linux 8.9 compared with CentOS 7.9. 

We investigated this performance discrepancy further and this article will reveal why there was a performance differential between the two operating systems.

## Resolution

The reason for the performance differences from both operating systems, was due to the BIOS settings on the HPE Blade machines that were used for testing. The CPU governor settings were set to `balanced power saving` mode. 

Upon configuring the CPU governor to `static high performance` mode in the BIOS, we saw a dramatic increase in performance for Rocky Linux 8.9.

Running the `futex` tests again on both Rocky Linux 8.9 and CentOS 7.9, produced the performance results as expected.

We also advise setting the `tuned-adm` profile to `hpc-compute` to ensure maximum performance from Rocky Linux.

## Conclusion

When comparing performance between a Rocky Linux X.X version and another operating system, CIQ advises to check the BIOS settings are set to have the CPU governor configured for maximum performance.

In addition, we advise setting the `tuned-adm` profile to `hpc-compute` to ensure maximum performance from Rocky Linux.

## References

* Futex Testing from the Linux Kernel Docs Team: https://www.kernel.org/doc/readme/tools-testing-selftests-futex-README

* General Mutex Subsystem from the Linux Kernel Docs Team: https://docs.kernel.org/locking/mutex-design.html

* tuned-adm man page: https://linux.die.net/man/1/tuned-adm

Performance Differences Between CentOS 7.9 and Rocky Linux 8.9 - a Root Cause Analysis and Recommendations


## Introduction
As mentioned [here](https://warewulf.org/docs/main/contents/containers.html#): 

*Since the inception of Warewulf over 20 years ago, Warewulf has used the model of the “Virtual Node File System” (VNFS) as a template image for the compute nodes. This is similar to a golden master image, except that the node file system exists within a directory on the Warewulf control node (e.g. a chroot()).*

*In hindsight, we’ve been using containers all along, but the buzzword just didn’t exist. Over the last 5-6 years, the enterprise has created a lot of tooling and standards around defining, building, distributing, securing, and managing containers, so Warewulf v4 now integrates directly within the container ecosystem to facilitate the process of VNFS image management.*

*If you are not currently leveraging the container ecosystem in any other way, you can still build your own chroot directories and use Warewulf as before.*

*It is important to understand that Warewulf is not running a container runtime on cluster nodes. While it is absolutely possible to run containers on cluster nodes, Warewulf is provisioning the container image to the bare metal and booting it. This container will be used as the base operating system and, by default, it will run entirely in memory. This means that when you reboot the node, the node retains no information about Warewulf or how it booted.*


## Problem
There could be cases where a node might not like to extract images for some reasons (BIOS settings, hardware specific problems, etc) and it is hard to find any solution to force iPXE for those nodes to pick the uncompressed images, and keep the other nodes using the compressed ones.

## Symptoms
When trying to provision the node with a compressed image, the server hangs during the PXE boot after downloading the container. After some minutes, the server might reboot with no warning message.

![Compressed Image](/images/articles/Compressedimage1.png)

## Resolution
There are two choices you can consider:
 
1. Probably the simpler method would be to copy `default.ipxe` to `nocompress.ipxe`, update it appropriately, and then direct nodes to use it with the ipxe template node attribute.
2. Define a node tag (something like `ipxe_nocompress`) and look for it in `default.ipxe`. You could then control whether to use compressed images or not based on what’s defined.


Pxe Boot with Compressed Container Images


## Introduction

When running a server that requires high availability, a user may not have the luxury to perform a reboot after making a configuration change.

The following guide will explain how to rename a network interface, without the need for a reboot.

## Problem

To change the names of network interfaces without rebooting the server.

## Resolution

### Prerequisites

Access to the `root` user account or a user with escalated privileges.

### Method 1 - ip link set

To temporarily change the name of an interface, use the `ip link set` command.

Bring the network interface that you wish to rename down:

```bash
sudo ip link set <INTERFACE_NAME> down
```

Rename the interface:

```bash
sudo ip link set <INTERFACE_NAME> name <NEW_INTERFACE_NAME>
```

Enable the newly renamed interface:

```bash
sudo ip link set <NEW_INTERFACE_NAME> up
```

**Please note your changes will be lost if you reboot your server.**

### Method 2 - udev rules by MAC Address

Check under `/etc/udev/rules.d/` that you have a `70-persistent-net.rules` file. If the file is not there, create it with `touch /etc/udev/rules.d/70-persistent-net.rules`

Confirm the interface you wish to edit by using `ip -brief address show`. You will see a similar output as in the below example:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# ip -brief address show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
enp8s0           UP             <PUBLIC_IP>/24
```

Open `/etc/udev/rules.d/70-persistent-net.rules` with your text editor of choice and modify the `NAME` field of the interface you wish to change. A before and after example follows:

Before:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="<PUBLIC_IP_HERE>", NAME="enp8s0"
```

After:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="<PUBLIC_IP_HERE>", NAME="new-name-1"
```

Bring the network interfaces down with `ip link set <INTERFACE_NAME> down`. In the author's case, the interface was `enp8s0`:

```bash
ip link set enp8s0 down 
```

Use the `udevadm control` command to reload the udev rules and request device events from the kernel with `udevadm trigger`:

```bash
udevadm control --reload-rules; udevadm trigger --subsystem-match=net --action=add
```

Confirm that the network interface was renamed with `ip -brief address show`. The author correctly found this to be the case:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# ip -brief address show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
new-name-1       DOWN           <PUBLIC_IP>/24
```

Bring the link back up using `ip link set <NEW_INTERFACE_NAME> up`. Please see the following example:

```bash
ip link set new-name-1 up 
```

Observe that the interface is up and has been renamed without requiring a reboot:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# ip -brief address show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
new-name-1       UP             <PUBLIC_IP>/24
```

Even if the machine is rebooted, the changes will still be present.

## Conclusions

With the methods above, you should be able to rename a network interface and avoid having to reboot.

## References & related articles

[udevadm man page](https://linux.die.net/man/8/udevadm)

[udev rules introduction](https://opensource.com/article/18/11/udev)


Renaming Network Interfaces Without Rebooting


## Introduction
This article addresses the issue where the Sriov configuration script runs manually but fails when executed automatically by systemd. The root cause is identified as a timing or dependency issue.

## Problem
The script `/etc/sriov/setup_dpdk_nic.sh` works when executed manually after a node restart but fails when run automatically by systemd. This indicates a potential timing or dependency issue with the service startup.

## Symptoms
- Script executes successfully when run manually after node restart.
- Script fails when triggered automatically by systemd during the boot process.

## Resolution
Adjust the service unit file to address potential timing and dependency issues. Update the service unit file as follows:

```ini
[Unit]
Description=Sriov Config Service
After=network-online.target systemd-udevd.service
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/etc/sriov/setup_dpdk_nic.sh
RemainAfterExit=yes
TimeoutStopSec=90
#StartLimitInterval=120
#StartLimitBurst=3

[Install]
WantedBy=multi-user.target
```

### Explanation of Changes:
1. **[Unit] Section:**
   - **After=network-online.target systemd-udevd.service:** Ensures the service starts only after the network is fully online and systemd-udevd is running. This is crucial for NIC-related configurations.
   - **Wants=network-online.target:** Ensures the network is available when the service runs.

2. **[Service] Section:**
   - **RemainAfterExit=yes:** Keeps the service active after the script completes, aiding in tracking and status reporting.
   - **TimeoutStopSec=90:** Sets a timeout for the stop operation.
   - **StartLimit* Parameters:** Commented out as they control service restart limits, which are less relevant for a oneshot service.

### Benefits:
- Ensures the NIC configuration script runs at the correct time during the boot process.
- Improves service status reporting by keeping the service active after script execution.

### Troubleshooting:
If the issue persists, provide the output of the following command for further review:
```sh
journalctl -u sriov-configure.service
```

## Root Cause
The problem likely stemmed from the script running before the network was fully online. By adjusting the service dependencies, we ensure the script runs at the appropriate time during the boot sequence.

Resolving Timing and Dependency Issues in Systemd Service Unit for Sriov Configuration


## Introduction
In certain environments, managing user namespace mappings across multiple hosts is essential for consistent access control and permissions. A common approach is to symlink /etc/subuid and /etc/subgid files to a shared storage location to easily distribute these mappings. However, this approach can lead to issues with tools like Apptainer that utilize user namespaces, as they rely on newuidmap and newgidmap utilities that do not support symlinks. This article outlines the problem, details the root cause, and offers alternative solutions to achieve synchronization across hosts without symlinking.

## Problem
To simplify the management of `/etc/subuid` and `/etc/subgid`, you may attempt to symlink these files to a shared storage. This approach is intended to ensure consistent mapping across multiple hosts.

## Symptoms
When attempting to build with Apptainer in fakeroot mode while using symlinked `subuid` and `subgid` files, you may encounter the following error:

```
# apptainer build --fakeroot rockylinux_fakeroot.sif docker://rockylinux:9
ERROR  : 'newgidmap' execution failed. Check that 'newgidmap' is setuid root or has setcap cap_setgid+eip.
ERROR  : Error while waiting event for user namespace mappings: no event received
```

## Root Cause
This issue occurs because `newuidmap` and `newgidmap` utilities do not support symbolic links. When these programs attempt to read the `subuid` and `subgid` files, they fail due to the `O_NOFOLLOW` flag in the `openat` system call, which prevents following symbolic links. The following strace output illustrates this behavior:

```
openat(AT_FDCWD, "/etc/subgid", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW) = -1 ELOOP (Too many levels of symbolic links)
```
The `O_NOFOLLOW` flag, as specified in the `open` man page, causes the call to fail with an _ELOOP_ error if the final path component is a symbolic link, which is what happens here.

## Resolution
To synchronize `subuid` and `subgid` files across hosts without using symlinks, consider alternative methods such as:

1. **Rsync and Cron Jobs**: Use `rsync` in conjunction with a cron job to periodically sync these files across hosts. For details, refer to the [Rocky Linux Rsync Guide](https://docs.rockylinux.org/guides/backup/rsync_ssh/) and [Rocky Linux Cron Guide](https://docs.rockylinux.org/guides/automation/cron_jobs_howto/).

2. **Ansible Automation**: Automate file synchronization with Ansible for a more flexible and scalable solution. Ascender, an orchestration tool from CIQ, can assist in managing these automation tasks. For more details, refer to the [Rocky Linux Ansible Guide](https://docs.rockylinux.org/books/learning_ansible/02-advanced/) and [Ascender Automation](https://ciq.com/products/ascender/).

## References & related articles
Rocky Linux Rsync Guide [https://docs.rockylinux.org/guides/backup/rsync_ssh/]  
Rocky Linux Cron Guide [https://docs.rockylinux.org/guides/automation/cron_jobs_howto/]  
Rocky Linux Ansible Guide [https://docs.rockylinux.org/books/learning_ansible/02-advanced/]  
Ascender Automation [https://ciq.com/products/ascender/]  

Error When Symlinking Subuid And Subgid


## Introduction

At CIQ, customers will often create tickets due to a kernel crash they experienced. As part of the investigation process, the Support Team asks for a `vmcore` dump to be uploaded to CIQ's Secure File Exchange (SFE) service for analysis.

For more information on how to upload a `vmcore` dump or other files to the SFE service, please consult the [How to Upload Files to the CIQ Secure File Exchange (SFE) Service](https://kb.ciq.com/article/how-load-files-using-ciq-secure-file-exchange) KB article.

This article will detail some of the steps that can be taken with the Crash Utility for `vmcore` dump analysis.

For further reading, the author highly recommends the following resources:

* [Crash Utility White Paper by David Anderson](https://crash-utility.github.io/crash_whitepaper.html)

* [Linux Kernel Crash Book - Everything you need to know by Igor Ljubuncic](https://www.dedoimedo.com/computers/www.dedoimedo.com-crash-book.pdf)

## Example issue

A customer created a ticket with CIQ, after their CentOS 7.9 node using CIQ Bridge had a kernel panic and rebooted. What follows will be how to setup the Crash Utility and the steps taken to debug the issue.

### Crash Utility environment setup

* Set up a Rocky Linux 9.x installation on a node size of your choice.

* Update all packages:

```bash
sudo dnf update -y
```

* Install the `kexec-tools` package:

```bash
sudo dnf install -y kexec-tools 
```

* From the node that went into a kernel panic state, find the kernel version:

```bash
uname -r
```

* With the kernel version identified, search and download the equivalent version of the `kernel-debuginfo-common` package. In the CentOS 7.9 example listed above, the kernel version was `3.10.0-1160.119.1.el7.x86_64`, thus the required package was `kernel-debuginfo-common-x86_64-3.10.0-1160.119.1.el7.x86_64.rpm`.

* Similarly, the same version of the `kernel-debuginfo` package is also needed. Again referencing the CentOS 7.9 example, downloading the `kernel-debuginfo-3.10.0-1160.119.1.el7.x86_64.rpm` package is required.

* **Note for Rocky Linux users** - to find the appropriate `kernel-debuginfo-common` and `kernel-debuginfo` packages, go to [The Rocky Linux Vault](https://dl.rockylinux.org/vault/rocky/). Using Rocky Linux 9.5 with the x86_64 architecture as an example, the `kernel-debuginfo-common` package is found [in the devel repository](https://dl.rockylinux.org/vault/rocky/9.5/devel/x86_64/os/Packages/k/) and `kernel-debuginfo` [is in the devel repository under debug](https://dl.rockylinux.org/vault/rocky/9.5/devel/x86_64/debug/tree/Packages/k/).

* Install both `kernel-debuginfo-common` and `kernel-debuginfo` packages using the `localinstall` command. The order the packages are installed in also matters:

```bash
dnf localinstall -y <path_to_kernel-debuginfo-common_rpm>
dnf localinstall -y <path_to_kernel-debuginfo_rpm>
```

* **Note for CentOS users** - the `vmcore` dump will be generated as a `vmcore.flat` file. This is not natively readable by the Crash Utility and thus has to be converted into a `vmcore` dump file. Perform the following command to convert the file:

```bash
/usr/sbin/makedumpfile -R ./vmcore < ./vmcore.flat
```

* Finally, start the Crash Utility by setting the `vmlinux` version as an argument under your `/usr/lib/debug/lib/modules/<kernel_version>` directory and pointing to the `vmcore` dump file:

```bash
crash /usr/lib/debug/lib/modules/<kernel_version>/vmlinux <path_to_vmcore_dump>
```

### Example crash dump analysis

* Starting the Crash Utility will bring up an overview of the `vmcore` dump, including the date the dump was taken, node name, memory availability, kernel version, and the kernel panic reason.

* An example kernel panic that CIQ often sees from customers running Kubernetes is `blocked tasks`:

```bash
"Kernel panic - not syncing: hung_task: blocked tasks"
```

* From the initial overview, the amount of memory available was plentiful during the kernel panic. However, as you will see later on in the article, having plentiful memory does not rule out a memory-related issue:

```bash
MEMORY: 1011.7 GB
```

* For deeper analysis on memory usage and availability, the `kmem -i` command can be ran and produces an output similar to the below example:

```bash
crash> kmem -i
                 PAGES        TOTAL      PERCENTAGE
    TOTAL MEM  261010406     995.7 GB         ----
         FREE  228447706     871.5 GB   87% of TOTAL MEM
         USED  32562700     124.2 GB   12% of TOTAL MEM
       SHARED  1626752       6.2 GB    0% of TOTAL MEM
      BUFFERS     3277      12.8 MB    0% of TOTAL MEM
       CACHED  20211378      77.1 GB    7% of TOTAL MEM
         SLAB   765701       2.9 GB    0% of TOTAL MEM

   TOTAL HUGE        0            0         ----
    HUGE FREE        0            0    0% of TOTAL HUGE

   TOTAL SWAP  1048575         4 GB         ----
    SWAP USED        0            0    0% of TOTAL SWAP
    SWAP FREE  1048575         4 GB  100% of TOTAL SWAP

 COMMIT LIMIT  131553778     501.8 GB         ----
    COMMITTED  26620006     101.5 GB   20% of TOTAL LIMIT
```

* Running the `sys -i` command shows more details about the hardware, as well as BIOS information, and product tags. Sensitive information such as serial numbers have been removed in the following example:

```bash
crash> sys -i
        DMI_BIOS_VENDOR: HPE
       DMI_BIOS_VERSION: A43
          DMI_BIOS_DATE: 12/03/2021
         DMI_SYS_VENDOR: HPE
       DMI_PRODUCT_NAME: ProLiant DL325 Gen10 Plus
    DMI_PRODUCT_VERSION:
     DMI_PRODUCT_SERIAL: -
       DMI_PRODUCT_UUID: -
       DMI_BOARD_VENDOR: HPE
         DMI_BOARD_NAME: ProLiant DL325 Gen10 Plus
      DMI_BOARD_VERSION:
       DMI_BOARD_SERIAL: -
    DMI_BOARD_ASSET_TAG: -
     DMI_CHASSIS_VENDOR: HPE
       DMI_CHASSIS_TYPE: 23
    DMI_CHASSIS_VERSION:
     DMI_CHASSIS_SERIAL: -
  DMI_CHASSIS_ASSET_TAG: -
```

* The `bt` command produces a backtrace output for the process that caused the kernel panic. The succeeding example shows process `650` running command `khungtaskd` was responsible for the kernel panic:

```bash
crash> bt
PID: 650      TASK: ffff9b2ffbb59080  CPU: 47   COMMAND: "khungtaskd"
 #0 [ffff9b2ffb663cb0] machine_kexec at ffffffffb2c69854
 #1 [ffff9b2ffb663d10] __crash_kexec at ffffffffb2d29f12
 #2 [ffff9b2ffb663de0] panic at ffffffffb33ab713
 #3 [ffff9b2ffb663e60] watchdog at ffffffffb2d56bfe
 #4 [ffff9b2ffb663ec8] kthread at ffffffffb2ccb621
```

* More information on `khungtaskd` can be found [at this article](https://www.tencentcloud.com/document/product/213/41974), however a summary can be observed beneath:

> The kernel hung task is based on a single kernel thread named as khungtaskd, which monitors processes in the TASK_UNINTERRUPTIBLE status. If a process stuck in D state during the period specified by kernel.hung_task_timeout_secs (defaults to 120 seconds), the stack information of this hung task process will be printed.

* To verify that the kernel is configured to crash in a `khungtaskd` state, the `p` command (to print the value of an expression) with the `sysctl_hung_task_panic` argument can be used:

```bash
crash> p sysctl_hung_task_panic
sysctl_hung_task_panic = $1 = 1
```

* Similarly the timeout can also be checked using the `p sysctl_hung_task_timeout_secs` command:

```bash
crash> p sysctl_hung_task_timeout_secs
sysctl_hung_task_timeout_secs = $2 = 360
```

* To view more information about a particular process, the `ps` command plus the PID number as an argument can be used:

```bash
crash> ps 650
      PID    PPID  CPU       TASK        ST  %MEM      VSZ      RSS  COMM
>     650       2  47  ffff9b2ffbb59080  RU   0.0        0        0  [khungtaskd]
```

* Finding out more information about which tasks were blocked, can be done with the `log` command and a `grep` for the word `INFO`. In the subsequent example, `systemd` is seen as being blocked:

```bash
crash> log | grep INFO
[3970064.996286] INFO: task systemd:1 blocked for more than 360 seconds.
```

* For showing the stack traces of all blocked tasks in a `TASK_UNINTERRUPTIBLE` state, the `foreach UN bt` command is recommended:

```bash
foreach UN bt
```

* Further filtering is also possible on all blocked tasks with this command string. The command picks tasks #2, #3, and #4 from each process, places them into three columns, counts the amount of stack trace messages occurrences and sorts from highest to lowest:

```bash
foreach UN bt | awk '/#[2-4] /{print $3,$5}' | paste - - - | sort | uniq -c | sort -nr
```

* An example is here:

```bash
crash> foreach UN bt | awk '/#[2-4] /{print $3,$5}' | paste - - - | sort | uniq -c | sort -nr
     46 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     proc_cgroup_show ffffffffb2d35146
     33 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     cgroup_lock_live_group ffffffffb2d2ed49
      2 __mutex_lock_killable_slowpath ffffffffb33b6520 mutex_lock_killable ffffffffb33b65fe    iterate_dir ffffffffb2e71d00
      1 schedule_timeout ffffffffb33b5831       wait_for_completion ffffffffb33b805d    wait_rcu_gp ffffffffb2cc828e
      1 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     proc_cgroupstats_show ffffffffb2d2f2be
      1 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     cgroup_release_agent ffffffffb2d2ffe5
      1 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     cgroup_free_fn ffffffffb2d34308
```

* From the above output, a high volume of `mutex_lock_slowpath` messages can be viewed. Researching the error attributes this to either a hardware issue or low resource availability (CPU / Memory / I/O).

* The full list of unique blocked tasks can be found by running `foreach UN bt | grep COMMAND | awk '{print $NF}' | sort -u`:

```bash
crash> foreach UN bt | grep COMMAND | awk '{print $NF}' | sort -u
"cadvisor"
"dockerd"
"filebeat"
"kubelet"
"kworker/113:0"
"kworker/31:1"
"process-exporte"
"runc"
"systemd"
"systemd-journal"
```

* To dig further into a blocked task, use the `set` command with the PID as the argument. The ensuing example sets the focus on the blocked task `systemd`:

```bash
crash> set 1
    PID: 1
COMMAND: "systemd"
   TASK: ffff9a32db050000  [THREAD_INFO: ffff9a32dbb9c000]
    CPU: 42
  STATE: TASK_UNINTERRUPTIBLE
```

* Then run `bt -l` for more verbose backtrace messages. In the above-mentioned `mutex_lock_slowpath` example, the process that was running before the `mutex_lock` was engaged was `proc_cgroup_show`:

```bash
crash> bt -l
#2 [ffff9a32dbb9fda8] __mutex_lock_slowpath at ffffffffb33b6ca7
    /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/include/linux/spinlock.h: 337
#3 [ffff9a32dbb9fe08] mutex_lock at ffffffffb33b602f
    /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/arch/x86/include/asm/current.h: 14
#4 [ffff9a32dbb9fe20] proc_cgroup_show at ffffffffb2d35146
    /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/kernel/cgroup.c: 4706
```

* The line number from the `cgroup.c` file being referenced by `proc_cgroup_show` was `4706`. It is possible to view the contents of the source code in the `vmcore` dump with the `list` or `l` command:

```bash
crash> l /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/kernel/cgroup.c: 4706
4701
4702            retval = 0;
4703
4704            mutex_lock(&cgroup_mutex);
4705
4706            for_each_active_root(root) {
4707                    struct cgroup_subsys *ss;
4708                    struct cgroup *cgrp;
4709                    int count = 0;
4710
```

* You can see from line `4704` that the `mutex_lock` is related to the `Cgroup` system. From the [kernel.org documentation on Cgroups](https://docs.kernel.org/admin-guide/cgroup-v1/cgroups.html):

> There is a global mutex, cgroup_mutex, used by the cgroup system. This should be taken by anything that wants to modify a cgroup. It may also be taken to prevent cgroups from being modified, but more specific locks may be more appropriate in that situation.

* Since `Cgroups` handle the allocation of resources, the mutex lock observed here points towards a lack of resources on the node.

* As an aside, if the `mutex_lock` code pointed to `mutex_lock(&dentry->d_inode->i_mutex);`, that would highlight an inode-related error on the storage medium and further investigation would be required there.

* Finally, after running the `log` command to dump the `system message buffer`, multiple `Hardware Error` messages were observed related to the customer's memory DIMMs:

```bash
crash> log
<output_redacted>
[3968484.124456] {176586}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 20480
[3968484.124460] {176586}[Hardware Error]: It has been corrected by h/w and requires no further action
[3968484.124461] {176586}[Hardware Error]: event severity: corrected
[3968484.124463] {176586}[Hardware Error]:  Error 0, type: corrected
[3968484.124464] {176586}[Hardware Error]:  fru_text: DIMM# Sourced
[3968484.124465] {176586}[Hardware Error]:   section_type: memory error
[3968484.124466] {176586}[Hardware Error]:   error_status: 0x0000000000040400
[3968484.124468] {176586}[Hardware Error]:   physical_address: 0x000000f877d52000
[3968484.124470] {176586}[Hardware Error]:   node: 0 card: 3 module: 1 rank: 2 bank: 6 row: 57311 column: 0
[3968484.124471] {176586}[Hardware Error]:  Error 1, type: corrected
[3968484.124472] {176586}[Hardware Error]:  fru_text: DIMM# Sourced
[3968484.124473] {176586}[Hardware Error]:   section_type: memory error
[3968484.124474] {176586}[Hardware Error]:   error_status: 0x0000000000040400
[3968484.124475] {176586}[Hardware Error]:   physical_address: 0x000000f7f9b63180
[3968484.124477] {176586}[Hardware Error]:   node: 0 card: 3 module: 1 rank: 2 bank: 6 row: 56806 column: 16
[3968484.124501] [Hardware Error]: Corrected error, no action required.
[3968484.125383] [Hardware Error]: CPU:0 (17:31:0) MC1_STATUS[-|CE|-|-|AddrV|-|-|-|-]: 0x940000000000009f
[3968484.126221] [Hardware Error]: Error Addr: 0x000000f877d52000, IPID: 0x0000000000000000
[3968484.126907] [Hardware Error]: Instruction Fetch Unit Extended Error Code: 0
[3968484.127593] [Hardware Error]: Instruction Fetch Unit Error: microtag probe port parity error.
[3968484.128280] [Hardware Error]: cache level: L3/GEN, tx: RESV
```

## Example issue resolution

Ultimately it was found that the lack of resources were due to `Hardware Errors` in the customer's memory DIMMs, thus causing a contention for resources resulting in tasks being blocked for over 360 seconds and therefore invoking a kernel panic.

## Conclusion

`vmcore` dump analysis is a vast and complicated process. There are almost a limitless number of ways that the kernel can crash and the root cause may not always be the same from system to system. The author hopes that the example analysis above for blocked tasks can help to demystify some of the research required and make it easier to identify a root cause if you run into a similar situation.

Tips and Tricks for Analyzing a vmcore dump with the Crash Utility


## Introduction

If no certificates are supplied by the Administrator during the configuration and installation
procedure, the Fuzzball Operator will create self-signed certificates for hosting Fuzzball
Orchestrate.  

## Problem

You may run into issues using the Fuzzball CLI if the cluster (context) you are accessing is using
a self-signed certificate.

## Symptoms

If you receive an error like the following when attempting to use the Fuzzball CLI it suggests that
your environment is not properly configured to recognize the self-signed TLS certificates in use.

```text
tls: failed to verify certificate: x509: certificate signed by unknown authority
```

## Resolution

An administrator can use the following commands to export the relevant certificates from the
Fuzzball K8s installation:

```text
# mkdir certs

# kubectl get secret -n cert-manager root-ca-cert -o "jsonpath={.data['ca\.crt']}" | base64 --decode >certs/ca.crt

# kubectl get secret -n cert-manager root-ca-cert -o "jsonpath={.data['tls\.crt']}" | base64 --decode >certs/tls.crt
```

These certificates can be distributed to the systems where the Fuzzball CLI is installed and then
used by adding an environment variable like so:

```text
$ export SSL_CERT_DIR=/path/to/certs
```

Alternately, CLI users can set an environment variable, `FUZZBALL_INSECURE=true` or use the
`--insecure` flag when invoking the Fuzzball command.

## Notes

The solutions selected here are appropriate for development clusters, testing, and debugging, but it
is ultimately more secure for administrators to use certificates issued by 3rd parties.  The
Fuzzball documentation contains a section describing how to do this.


Using the CLI with Self-Signed TLS Certificates


## Introduction
As explained in this [guide](https://warewulf.org/docs/main/contents/provisioning.html), the only thing that Warewulf requires to provision is that the node is set to PXE boot. You may need to change the boot order if there is a local disk present and bootable. This is a configuration change you will have to make in the BIOS of the cluster node, as this is the first step of the provisioning process. 
Once the nodes boot to PXE, PXE will request a BOOTP/DHCP address on the network, the Warewulf controller’s DHCP server will respond with a network configuration and filename to try and boot, PXE will attempt to download the filename referred to in the DHCP response via TFTP. 
The downloaded file will execute an iPXE stack which will reach out to the Warewulf server for it’s configuration. Then, the Warewulf server will generate the iPXE configuration which will include directions of what else is necessary to download and how to boot. 
The kernel, container image, kernel modules, and system overlay are all downloaded over REST HTTP from the Warewulf Server. iPXE executes the kernel and processes the overlays to provide a unified root file system. 
Warewulf bootstraps the initialization of cluster node’s operating system: File System (re)configuration, SELinux, and `wwclient` is called as a background daemon and sleeps until network is ready. And for last, the Warewulf bootstrap execs the container’s `/sbin/init`


## Problem
Node(s) are not booting after set to boot from PXE.

## Resolution
The resolution for this problem is not straight forward as this problem could be caused by several different parts of the booting process. Therefore is good starting with a sanity check
 
* Ensure httpd, dhcpd, tftp-server.socket (or xinetd) are all enabled and running

`systemctl status <service>`

* Ensure firewall rules are set or disabled. Is good to stop firewall completely to rule out any firewall problems

`systemctl stop firewalld`

* Check Selinux. If Selinux is enforcing check [this guide](https://warewulf.org/docs/main/contents/security.html) to enable SELinux if needed, or just disable SELinux to rule out any SELinux issues.

`setenforce 0`

### Test TFTP Server.
Another option to check is making sure TFTP service is working properly. 
We need to make sure the server is able to get a file from warewulf directory
`tftp <tftpserverIPADDR> -c get /var/lib/warewulf/bin-x86_64-efi-snponly.efi`

**Note:** This usually only test the tftp server is working properly but to discard any network issue, we advice to run the command from a node in the same network as the warewulf headnode as well, and check if it gets the file successfuly from the tftp server.


## References & related articles
[Node Provisioning](https://warewulf.org/docs/main/contents/provisioning.html)


Troubleshooting PXE Booting, Node Provisioning.


## Introduction
This article addresses the issue of delayed errata page publication in the Rocky Linux repository.

## Problem
The errata page for a specific package update may not be immediately available after the package is made available in the repository. This delay can occur due to various administrative or procedural reasons within the maintenance and release management process.

## Resolution
If you encounter a delayed errata page publication, follow these steps:

1. **Check the package update history**: Verify that the package update was recently made available in the repository.
2. **Verify the errata page status**: Check the Rocky Linux forums or other community resources for information on the status of the errata page.
3. **Wait for the errata page to be published**: The errata page may be in the process of being updated or created. Wait for the page to be published before attempting to access it.
4. **Contact CIQ Support**: If the issue persists, contact CIQ Support for further assistance.

Note: The errata page publication process is owned by the open source community, and delays may occur due to various reasons. Be patient and allow sufficient time for the errata page to be published.

Troubleshooting Missing Errata Pages in Rocky Linux Repository


## Introduction
[Infiniband network](https://www.nvidia.com/en-us/networking/products/infiniband/) needs a "TYPE" value for the network script. 

As discussed in [this thread](https://github.com/warewulf/warewulf/issues/457) The warewulf version 4.3, does not have a `TYPE=""` in the overlay: https://github.com/warewulf/warewulf/blob/v4.3.0/overlays/wwinit/etc/sysconfig/network-scripts/ifcfg.ww#L12

## Problem
The Infiniband network fails to come up. 

## Resolution
The fix was added in [version 4.4.0](https://github.com/warewulf/warewulf/releases/tag/v4.4.0). 
Upgrade Warewulf to a newer version.


## References & related articles
[Deploying a Dell PowerEdge and Cornelis Omni-Path Cluster with Warewulf](https://ciq.com/blog/deploying-a-dell-poweredge-and-cornelis-omni-path-cluster-with-warewulf/)
[Infiniband network](https://www.nvidia.com/en-us/networking/products/infiniband/)


Warewulf Config for Infinibad Interfaces

<h2>Introduction</h2>
<p>A <code>mutex</code> (meaning <code>mutual exclusion</code>) prevents multiple threads from accessing the same resource simultaneously. When CIQ was running a <code>futex</code> test (a form of testing that checks the Linux kernel <code>futex</code> system call API), we observed <code>mutex</code> slowdown when running the <code>futex</code> testing on Rocky Linux 8.9 compared with CentOS 7.9.</p>
<p>We investigated this performance discrepancy further and this article will reveal why there was a performance differential between the two operating systems.</p>
<h2>Resolution</h2>
<p>The reason for the performance differences from both operating systems, was due to the BIOS settings on the HPE Blade machines that were used for testing. The CPU governor settings were set to <code>balanced power saving</code> mode.</p>
<p>Upon configuring the CPU governor to <code>static high performance</code> mode in the BIOS, we saw a dramatic increase in performance for Rocky Linux 8.9.</p>
<p>Running the <code>futex</code> tests again on both Rocky Linux 8.9 and CentOS 7.9, produced the performance results as expected.</p>
<p>We also advise setting the <code>tuned-adm</code> profile to <code>hpc-compute</code> to ensure maximum performance from Rocky Linux.</p>
<h2>Conclusion</h2>
<p>When comparing performance between a Rocky Linux X.X version and another operating system, CIQ advises to check the BIOS settings are set to have the CPU governor configured for maximum performance.</p>
<p>In addition, we advise setting the <code>tuned-adm</code> profile to <code>hpc-compute</code> to ensure maximum performance from Rocky Linux.</p>
<h2>References</h2>
<ul>
<li>
<p>Futex Testing from the Linux Kernel Docs Team: <a href="https://www.kernel.org/doc/readme/tools-testing-selftests-futex-README">https://www.kernel.org/doc/readme/tools-testing-selftests-futex-README</a></p>
</li>
<li>
<p>General Mutex Subsystem from the Linux Kernel Docs Team: <a href="https://docs.kernel.org/locking/mutex-design.html">https://docs.kernel.org/locking/mutex-design.html</a></p>
</li>
<li>
<p>tuned-adm man page: <a href="https://linux.die.net/man/1/tuned-adm">https://linux.die.net/man/1/tuned-adm</a></p>
</li>
</ul>
