
## Introduction

This article will cover assigning an IP address via DHCP using Warewulf, to a device that is not managed directly by Warewulf itself. This is useful in instances where you want to assign an address to a Network Attached Storage (NAS) device or similar non-compute node.

The implementation can be done by modifying the included DHCP template that comes with Warewulf.

## Resolution

First, backup the DHCP template (this is optional but recommended):

```bash
# For Warewulf 4.5.x
cp /var/lib/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww /var/lib/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.bak

# For Warewulf 4.6.x
cp /usr/share/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww /usr/share/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww.bak
```

There are two implementation methods available: adding the entries directly or creating a separate entry under `/etc/dhcp/hosts.conf` and including each host there. Both of these will be explained below.

### Option 1 - Add entries directly

You'll add each individual entry directly to the `dhcpd.conf` file. You will use the following entry as an example:

```bash
host MyDevice {
    hardware ethernet 00:11:22:33:44:55;
    fixed-address 10.0.0.5;
}
```

Edit the following template file `/var/lib/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww` for Warewulf 4.5.x using [vi](https://docs.rockylinux.org/books/admin_guide/05-vi/) or the text editor of your choice.

Alternatively for Warewulf 4.6.x, edit this template file instead: `/usr/share/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww`

 At the bottom of the `dhcpd.conf.ww` file, we are looking for the line `{{/* dhcp enabled */}}`.

 Move this under the `{{- end}}` line.

 In between the `{{/* dhcp enabled */}}` and `{{- end}}` lines, add your host entry, similar to the example below:

```bash
...
{{end -}}{{/* range NetDevs */}}
{{end -}}{{/* range AllNodes */}}
{{end -}}{{/* if static */}}
{{- else}}
{{abort}}
{{- end}}

host MyDevice {
    hardware ethernet 00:11:22:33:44:55;
    fixed-address 10.0.0.5;
}

{{/* dhcp enabled */}}
```

Add as many hosts as required.

Run the `wwctl configure dhcp` command to update the DHCP config.

Verify your hosts were added correctly using `cat /etc/dhcpd.conf`.

### Option 2 - Include a separate hosts file

Instead of adding each host to the template individually, you can also create a separate file and then include this in the main template.

Create a `hosts.conf` file under `/etc/dhcp/`.

Edit the `hosts.conf` file and add each individual host. Please see example host additions below:

```bash
host MyDevice {
    hardware ethernet 00:11:22:33:44:55;
    fixed-address 10.0.0.5;
}

host MyOtherDevice {
    hardware ethernet AA:BB:CC:DD:EE:FF;
    fixed-address 10.0.0.6;
}
```

Now edit the template.

Similar to "Option 1 - add entries directly", we are looking for the `{{/* dhcp enabled */}}` line in the file.

Move the `{{/* dhcp enabled */}}` line under the `{{- end}}` line.

In the middle of the `{{/* dhcp enabled */}}` and `{{- end}}` lines, add your host information.

Edit the `dhcpd.conf.ww` template file.

For Warewulf 4.5.x, this file is located under `/var/lib/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww`.

For Warewulf 4.6.x, this file is found under `/usr/share/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww`.
Exactly as before, you want to find the `{{/* dhcp enabled */}}` line.

Move this under the `{{- end}}` line.

Add your newly created `hosts.conf` via the `Include` line between the `{{- end}}` and `{{/* dhcp enabled */}}` lines. Below is an example:

```bash
...
{{end -}}{{/* range NetDevs */}}
{{end -}}{{/* range AllNodes */}}
{{end -}}{{/* if static */}}
{{- else}}
{{abort}}
{{- end}}

{{ Include "/etc/dhcp/hosts.conf" }}

{{/* dhcp enabled */}}
```

Again, run the `wwctl configure dhcp` command to update the DHCP config.

You can verify your hosts were added by checking the output of the `cat /etc/dhcpd.conf` command.

## Conclusion

You have successfully assigned an IP address to a device via DHCP using Warewulf, therefore opening a world of possibilities to assign IPs to any device on your network; not just compute nodes. This shows just how flexible Warewulf is as a DHCP server, as well as for cluster management.

## References & related articles

VI Text Editor: [Link](https://docs.rockylinux.org/books/admin_guide/05-vi/)  
Warewulf Configuration: [Link](https://warewulf.org/docs/main/contents/configuration.html#warewulf-conf)  


How to Add a Static DHCP Entry into Warewulf 4.5.x and 4.6.x


## Introduction

This guide will have you setup and run Ascender on a K3s cluster.

## Problem

Many customers face difficulties when managing their infrastructure at scale. This is where Ascender comes in to ensure scalable patching, security hardening and more across your fleet.

## Symptoms

A new user may encounter difficulties installing Ascender onto their K3s cluster.

## Resolution

### Prerequisites

* Same OS family as Rocky Linux. Rocky Linux 8.x or Rocky Linux 9.x is recommended as the base OS.

* System requirements:

  * CPUs: 2

  * Memory: 8GB (if installing both Ascender and Ledger)

  * Disk: 20GB (if installing both Ascender and Ledger)

* Internet access for the K3s cluster.

* Perform the below steps with a user that has privilege escalation. Do not use the `root` user account.

### Setup

* Update the node running K3s:

```bash
sudo dnf update -y
```

* Install `git`:

```bash
sudo dnf install -y git
```

* Allow these ports in `firewalld`:

```bash
sudo firewall-cmd --permanent --zone=public --add-port=22/tcp
sudo firewall-cmd --permanent --zone=public --add-port=80/tcp
sudo firewall-cmd --permanent --zone=public --add-port=443/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5432/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5895/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5986/tcp
sudo firewall-cmd --reload
```

* Clone the Ascender repository:

```bash
git clone https://github.com/ctrliq/ascender-install.git
```

* Navigate to the installation directory:

```bash
cd ascender-install
```

* Run the `config_vars.sh` script:

```bash
./config_vars.sh
```

* The script will present you with multiple questions. Answer each of them according to how you want to set up your Ascender environment and if you already have K3s installed on your node.

* Run the setup script to install Ascender:

```bash
sudo ./setup.sh
```

* Once complete, the final playbook recap will look like the following example:

```bash
PLAY RECAP *************************************************************************************************************************
ascender_host              : ok=14   changed=6    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0
localhost                  : ok=72   changed=27   unreachable=0    failed=0    skipped=4    rescued=0    ignored=0

ASCENDER SUCCESSFULLY SETUP
```

### Accessing Ascender

* You can access Ascender directly via the internal CLUSTER IP address.

## Notes

* To monitor the Ascender setup on your K3s cluster during install:

```bash
kubectl get pod -n ascender -w
```

* For further details surrounding particular pods:

```bash
kubectl describe pod -n ascender [pod name]
```

## References & related articles

[Ascender General Prerequisites](https://github.com/ctrliq/ascender-install/tree/main?tab=readme-ov-file#general-prerequisites)
[Ascender K3s Install Guide](https://github.com/ctrliq/ascender-install/blob/main/docs/k3s/README.md)


Quick Start Guide for Installing Ascender on K3s


## Introduction

This guide explains how to synchronize repositories locally using `reposync`. It covers syncing both publicly available repositories and private repositories, such as those within CIQ Depot. This guide will provide you with two options for downloading packages depending on the complexity of your needs.

## Problem

You need to create a local copy of a repository for an air-gapped environment, a network with limited or slow internet access, or to provide faster access to a local set of servers.

## Prerequisites

A web server is required to serve the downloaded packages. You can follow our [guide](https://ciq.com/blog/how-to-set-up-a-web-server-in-rocky-linux/) on how to set up a web server. When syncing repositories, you will also need to consider the storage requirements.

For smaller repositories that only feature updates such as our [Long Term Support (LTS)](https://ciq.com/services/long-term-support/) repository, the overall size can be somewhere in the range of 1GB to 15GB or more in size.

The full repository for Rocky Linux or CentOS can potentially exceed 100GB in total size.

In this guide, the web server will have an IP address of `192.168.1.30`.

## Option 1: Sync on an existing server

If you are using a Rocky Linux server or another server using CIQ's [CentOS Bridge service](https://ciq.com/products/ciq-bridge/) that is already configured for the repository you are looking to sync, you can leverage its existing setup to download the repository.

Identify the repository you want to download by running `dnf repolist`. The first column will provide you with the repo id that you will use in your reposync command.

In this example, CIQ is syncing using the `repoid` of *rocky-lts-9.2.x86_64* attached to our *CIQ LTS for Rocky Linux 9.2* repository.

```bash
dnf reposync -p repodata --download-metadata \
  --repoid=rocky-lts-9.2.x86_64 --download-path=/var/www/html/rocky-lts-9.2.x86_64/
```

If you are trying to sync our [CentOS Bridge](https://depot.ciq.com/my/bridge/) repository (requires a subscription to CIQ Depot) from a CentOS 7 server, the command is different:

```bash
reposync -p repodata --download-metadata \
   --repoid=ciq-bridge.x86_64 --download_path=/var/www/html/ciq-bridge/
```

The above commands will store the repository files in the local web server path `/var/www/html/`. You can also store these files on shared storage for centralized access or copy them to another server.

## Option 2: Sync on a separate server

If you want to run this on a dedicated server or need to sync multiple repositories, consider creating a dedicated repo file(s) to manage the information.

### Step 1: Create a repository configuration file

If you plan to sync CIQ Depot repositories, you can obtain the necessary repo information from the following location:

1. Navigate to [My Products](https://depot.ciq.com/my/).
2. Select the repository you would like to sync.
3. Click the button with the three dots on the right-hand side of the repository name.

4. Select **"DNF Repo Config"**.

For our example, we will use the publicly available Rocky Linux 9 BaseOS repository. We will create a new repository file named `rocky-baseos-9.repo` in our root directory. Feel free to save this in a more permanent location.

```ini
[rocky-baseos-9]
name=Rocky Linux 9 - BaseOS
baseurl=http://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/
gpgcheck=1
enabled=1
countme=1
metadata_expire=6h
gpgkey=https://dl.rockylinux.org/pub/rocky/RPM-GPG-KEY-Rocky-9
```

The repo id is specified at the top of the file between the `[]` characters. In this case, `[rocky-baseos-9]`.

Repeat the above step to create separate files for each repository you plan to sync.

Alternatively, you can add multiple repositories to a single file, ensuring each has a unique repo id.

### Step 2: Run `reposync`

Once the configuration file is created, you can run the `reposync` command as before. Only now you will specify the repo file like in the below example:

```bash
dnf -c ./rocky-baseos-9.repo reposync -v -p repodata \
--download-metadata --repoid=rocky-baseos-9 \
--download-path=/var/www/html/rocky-baseos-9/
```

Repeat this step for each file and/or repository you intend to sync ensuring that you change the file, repo id, and folder as needed.

### Step 3: Verify and configure clients

After all downloads are complete, confirm the repository is accessible via your web browser.

In the below example, our repository is located at `http://192.168.1.30/rocky-baseos-9/rocky-baseos-9/`.

Once you’ve verified that the packages are being served correctly, configure Rocky Linux to use the repository by updating the appropriate file in `/etc/yum.repos.d/`.

As illustrated below, since we are serving the BaseOS repository, we will modify `/etc/yum.repos.d/rocky.repo`.

```ini
[baseos]
name=Rocky Linux $releasever - BaseOS
#mirrorlist=https://mirrors.rockylinux.org/mirrorlist?arch=$basearch&repo=BaseOS-$releasever$rltype
baseurl=http://192.168.1.30/rocky-baseos-9/rocky-baseos-9/
gpgcheck=1
enabled=1
countme=1
metadata_expire=6h
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Rocky-9
```

## Notes

### Automate reposync

There are several ways to automate fetching the latest packages via `reposync`.

For a complete, centralized solution, we recommend running a playbook within [Ascender](https://ciq.com/products/ascender/).

Alternatively, you can set up a simple [cronjob](https://docs.rockylinux.org/guides/automation/cron_jobs_howto/) to pull the latest packages automatically. For example, you might create `/etc/cron.d/reposync` in the example provided:

```bash
# Run the hourly jobs
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
@daily root dnf -c /root/rocky-baseos-9.repo reposync -v -p repodata --download-metadata --repoid=rocky-baseos-9 --download-path=/var/www/html/rocky-baseos-9/
```

This will run the cronjob daily at midnight. Add a new line for each repository you intend to sync.

For more granular control of `cronjob` timing, please consult [this documentation on cron jobs](https://docs.rockylinux.org/guides/automation/cron_jobs_howto#more-complex-options).

### Delete old packages on sync

When running `reposync`, older packages will not be deleted by default.

This means your local repository will contain all versions of packages, as newer versions are released.

This is helpful if you desire having older packages available that are not retained in the upstream repository. However, this can take up significant storage space.

To delete packages locally that are no longer present on the remote server, append `--delete` to the `reposync` command like so:

```bash
dnf -c ./rocky-baseos-9.repo reposync -v -p repodata \
--download-metadata --repoid=rocky-baseos-9 \
--download-path=/var/www/html/rocky-baseos-9/ \
--delete
```

Although the old RPM files remain in the repository folders, they won’t be accessible via DNF on remote systems.

This is because the metadata is copied from upstream.

To retain these preexisting RPMs, update the metadata using a tool like [createrepo](https://github.com/rpm-software-management/createrepo).

## References & related articles

[How to Set Up a Web Server in Rocky Linux](https://ciq.com/blog/how-to-set-up-a-web-server-in-rocky-linux/)  
[Rocky Linux Cronjob Guide](https://docs.rockylinux.org/guides/automation/cron_jobs_howto/)  
[DNF RepoSync Options](https://dnf-plugins-core.readthedocs.io/en/latest/reposync.html)  
[DNF RepoManage Options](https://dnf-plugins-core.readthedocs.io/en/latest/repomanage.html)


How to Create a Local Mirror Using Reposync with CIQ Depot


## Introduction

Customers often need a custom ISO for a litany of reasons - perhaps they require a mirror of a repository or to edit kernel cmdline parameters for their own customers.

## Problem

This guide was created after a customer raised a question in [#1629 - creating a custom installation ISO](https://ciq.zendesk.com/agent/tickets/1629), asking for assistance on how to create a Rocky Linux 8.10 ISO. 

## Symptoms

The customer did not have clear direction or resources on how to build their own ISO.

## Resolution

* The customer was given multiple documentation sources to reference, a high level ISO generation guide and multiple commands to execute.
* A guide on how to create a custom Rocky Linux ISO was also pushed upstream to the Rocky Linux Docs. Please find the same guide below:

### Prerequisites

* A 64 bit machine running Rocky Linux 9 to build the new ISO image.
* A Rocky Linux 9 DVD ISO image.
* A `kickstart` file to apply to the ISO.
* Read the Lorax [Quickstart](https://weldr.io/lorax/lorax.html#quickstart) and [mkksiso](https://weldr.io/lorax/mkksiso.html) documentation to become familiar with how the `Anaconda` `boot.iso` is created.

### Package installation and setup

* Install the `lorax` package:
```
sudo dnf install -y lorax
```

### Building the ISO with a kickstart file

* Run the `mkksiso` command to add a `kickstart` file and then build a new ISO:
```
mkksiso --ks <PATH_TO_KICKSTART_FILE> <PATH_TO_ISO_TO_MODIFY> <OUTPUT_PATH_FOR_BUILT_ISO>
```
* Below is an example `kickstart` file `example-ks.cfg`, which sets up a Rocky Linux 9.5 `Server With GUI` environment:
```
lang en_GB
keyboard --xlayouts='us'
timezone Asia/Tokyo --utc
reboot
cdrom
bootloader --append="rhgb quiet crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
zerombr
clearpart --all --initlabel
autopart
network --bootproto=dhcp
firstboot --disable
selinux --enforcing
firewall --enabled
%packages
@^server-product-environment
%end
```

### Adding a repository with its packages to an ISO image

* Make sure the repository you want to add has the `repodata` directory inside of it. If not, this can be created using the `createrepo_c` command and this can be installed with `sudo dnf install -y createrepo_c`
* Add the repository to your `kickstart` file, using the following syntax:
```
repo --name=extra-repo --baseurl=file:///run/install/repo/<YOUR_REPOSITORY>/
```
* Add your repository using the `--add` flag with the `mkksiso` tool:
```
mkksiso --add <LINK_TO_YOUR_REPOSITORY> --ks <PATH_TO_KICKSTART_FILE> <PATH_TO_ISO_TO_MODIFY> <OUTPUT_PATH_FOR_BUILT_ISO>
```

* The process is detailed further using the `baseos` repository in the example below:
* The `base os` repository will be locally downloaded along with all of its packages:
```
dnf reposync -p ~ --download-metadata --repo=baseos
```
* Then add the repository to the `kickstart` file:
```
repo --name=extra-repo --baseurl=file:///run/install/repo/baseos/
```
* The `kickstart` file would look like the following:
```
lang en_GB
keyboard --xlayouts='us'
timezone Asia/Tokyo --utc
reboot
cdrom
bootloader --append="rhgb quiet crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
zerombr
clearpart --all --initlabel
autopart
network --bootproto=dhcp
firstboot --disable
selinux --enforcing
firewall --enabled
%packages
@^server-product-environment
repo --name=extra-repo --baseurl=file:///run/install/repo/baseos/
%end
```
* Then point the `mkkiso` command directly to the repository directory and build the ISO:
```
mkksiso --add ~/baseos --ks example-ks.cfg ~/Rocky-9.5-x86_64-dvd.iso ~/Rocky-9.5-x86_64-dvd-new.iso
```

### Conclusion

What has been discussed here are just a few of the options available to tweak and build your own Rocky Linux ISO. For further ways, including modifying the kernel cmdline arguments, the author highly recommends to go through the [mkksiso](https://weldr.io/lorax/mkksiso.html) documentation in more detail.


## References & related articles

[Anaconda Documentation](https://anaconda-installer.readthedocs.io/en/latest/)\
[Automating the Installation with Kickstart - Fedora Wiki](https://docs.fedoraproject.org/en-US/fedora/f36/install-guide/advanced/Kickstart_Installations/#chap-kickstart-installations)\
[Dracut - Fedora Wiki](https://fedoraproject.org/wiki/Dracut)\
[Lorax Documentation](https://weldr.io/lorax/)\
[mkksiso Documentation](https://weldr.io/lorax/mkksiso.html)\
[Mock Repository](https://github.com/rpm-software-management/mock)\
[Peridot Repository](https://github.com/rocky-linux/peridot)


Creating a Custom Rocky Linux ISO


## Introduction

This article is a great reference for customers looking to learn more about how CIQ evaluates and remediates security vulnerabilities. 

## FAQ 

**What CVE’s are addressed by CIQ Bridge and CIQ LTS?**  
For both Bridge and LTS subscriptions, CIQ focuses on resolving CVEs with CVSS scores of 7 and above. CIQ also considers customer impact and environmental factors.CIQ evaluates CVE’s taking into account the OS, the build environment and other Linux vendor scores to determine the applicability of the CVE to our distribution. CIQ prioritizes CVE’s based on CVSS score, package exposure, popularity, confidence in the potential fixes (vs. risk of them introducing bugs), and customer requests.  

**What does the CVSS Score mean?**  
CIQ evaluates scores from [NIST CVSS 3.x Scoring System](https://nvd.nist.gov/vuln-metrics/cvss), MITRE and other OS Vendors to evaluate and rank vulnerabilities in a standardized and repeatable way.  These scores are used as inputs for CIQ to determine the appropriate impact of the CVE to our distribution.

**Are there SLA’s for CVE remediation?**  
CIQ does not provide SLA’s for any CVE remediation. They are prioritized based on the criteria discussed above.

**Where can I see the status of a given CVE?**  
All fixes can be found in the CSAF files provided by the public [CIQ Advisories Repository](https://github.com/ctrliq/advisories). CVE fixes are also published in the [Mountain Portal](https://secure.ciq.com/).  For any other status updates, please reach out to your Customer Success Manager or open a Support ticket.

**How do CSAF files relate to the output of my scanning tool?**  
Scanners use CSAF files and data from OS vendors to determine if a CVE is fixed based on the version of the RPM. Scanner vendors are responsible for processing Rocky Linux and CIQ CSAF files and advisories to determine if a fix has been released for a given CVE.  If your scanner is not ingesting the CSAF files from Rocky/CIQ LTS it will produce false positives.

**How do CVE’s relate to Security Advisories?**  
Security advisories can contain one or more CVE’s. CIQ evaluates CVEs individually and not Security Advisories. CVE’s will be fixed based on the criteria mentioned above.  


CIQ CVE Remediation - Frequently Asked Questions


## Introduction

Many businesses need to have multiple desktop environments set up and to tweak them accordingly as per their requirements.

CIQ delved deep into one such test case, in order to have the MATE desktop environment installed alongside the GNOME desktop environment and to remove the ability to login to MATE from the GNOME Display Manager (GDM) login screen.

This article will describe how to remove the option to login to MATE from GDM.

## Problem

An administrator would like to disable the option to login to MATE or other alternative desktops from the GDM login window without uninstalling the desktop environment.

## Resolution

Simply move the `mate.desktop` file to another location or rename it to `mate.desktop.disabled`:

```bash
mv /usr/share/xsessions/mate.desktop /usr/share/xsessions/mate.desktop.disabled
```

After a reboot, you should no longer see the option at the GDM login screen to login to MATE when clicking the cog icon after entering your username.

## Conclusion

Following the steps listed in this article by renaming the `mate.desktop` file should help you better control the desktop environments (in this case with MATE) that users have access to from the GNOME Display Manager in your environment.

An example of where this is beneficial is with separating users into two groups: those needing an alternative desktop for specific applications and those that do not and can complete their assigned tasks using GNOME.

## References & related articles

Rocky Linux Desktops: [Link](https://docs.rockylinux.org/desktop/)  


How to Remove Access to the Mate Desktop Environment from the GNOME Desktop Manager Login Screen


## Introduction

This article provides a step-by-step guide to manually disable the screensaver lock screen setting on the GNOME desktop environment. Follow the instructions below to complete the process using `dconf` and `dconf-editor`.

## Problem

Users may need to disable the screensaver lock screen setting on their GNOME desktop environment for various reasons, such as preventing interruption during tasks.

## Prerequisites

Ensure that the `dconf` and `dconf-editor` packages are installed on your system. If they are not installed, use the following command to install them:

```bash
sudo dnf install -y dconf dconf-editor
```

## Steps to disable the screensaver lock-screen setting

1. **Launch dconf-editor**

   Open the `dconf-editor` by running the following command in your terminal:

   ```bash
   dconf-editor
   ```

   This will open the `dconf` GUI.

2. **Navigate to the screensaver settings**

   In the `dconf-editor`, navigate to the screensaver settings by following this path:

   ```text
   org > gnome > desktop > screensaver
   ```

3. **Edit lock screen setting**

   - Click on `lock-enabled`. This will open a screen where you can change the value.
   - Toggle the `Use default value` to off.
   - Change the `Custom value` to `false`.
   - Click the blue check-mark at the bottom to confirm your changes.

   ![dconf-editor screensaver settings](/images/articles/disable-screensaver-lock-screen/screensaver-dconf-editor_m.jpg)

4. **Handling errors**

   If you encounter the following error in your terminal while attempting to save the settings:

   ```plaintext
   (dconf-editor:17050): dconf-WARNING **: [time]: failed to commit changes to dconf: Failed to execute child process "dbus-launch" (No such file or directory)
   ```

   This indicates that the `dbus-launch` utility is missing. Resolve this by exiting the GUI and installing the `dbus-x11` package:

   ```bash
   sudo dnf install -y dbus-x11
   ```

   After installing, relaunch the GUI and attempt to save your changes again. A system reboot may be required if the error persists.

## Validate the changes

To verify that the lock screen has been disabled, run one of the following commands:

```bash
gsettings get org.gnome.desktop.screensaver lock-enabled
```

or

```bash
dconf read /org/gnome/desktop/screensaver/lock-enabled
```

These commands should return `false`, indicating that the lock screen has been disabled.

## Conclusion

By following these steps, you should be able to manually disable the screensaver lock screen setting on your GNOME desktop environment. If you encounter any issues, ensure that all required packages are installed and correctly configured.


Disabling the Screensaver Lock Screen Setting


## Introduction

As specified in the [Warewulf Disk Management documentation](https://warewulf.org/docs/v4.5.x/contents/disks.html#rocky-linux), packages for [ignition](https://coreos.github.io/ignition/) are not available for the Rocky Linux 8.x series. They are available for the Rocky Linux 9.x series as part of the `Appstream` repository.

## Symptoms 

When you try to install `gdisk` or `ignition` on a Rocky Linux 8.x system, you will find it is not possible, due to the packages being unavailable in the repositories:

```bash
[root@Rocky-Linux-8-10-Test-Machine ~]# dnf install ignition
Last metadata expiration check: 0:04:50 ago on Fri 27 Dec 2024 05:46:44 AM UTC.
No match for argument: ignition
Error: Unable to find a match: ignition
```

## Resolution

### Prerequisites

You will require access to the `root` account or a user with escalated privileges.

### Build Ignition from source

When creating your container, add the following lines to your configuration:

```bash
&& (cd /tmp && git clone https://github.com/coreos/ignition.git) \
&& (cd /tmp/ignition && make all && make install) \
```

### Example container creation with Warewulf and ignition on Rocky Linux 8.10

Current disk structure of the nodes:

```bash
[root@Rocky-Linux-8-10-Warewulf-Controller-Node ~]# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sr0     11:0    1  1024M  0 rom  
vda    253:0    0   180G  0 disk 
├─vda1 253:1    0   260M  0 part /boot/efi
└─vda2 253:2    0 179.8G  0 part /
```

The Warewulf Rocky Linux 8.10 container was used for testing:

```bash
wwctl container import docker://ghcr.io/warewulf/warewulf-rockylinux:8 rockylinux-8 --build
```

**The following commands were all ran from the Controller Node:**

Created a 1GB `boot` partition for the Compute Node with `ext4` and wiped the filesystem at boot:

```bash
wwctl node set --diskname /dev/vda --diskwipe --partname boot --partsize=1024 --partnumber 1 --partcreate --fsname boot --fsformat ext4 --fspath /boot --fswipe
```

Assigned the rest of the disk space to the root filesystem, formatted with `btrfs`, and wiped the disk at boot for the Compute Node:

```bash
wwctl node set --diskname /dev/vda --partname root --partnumber 2 --partcreate --fsname root --fsformat btrfs --fspath / --fswipe
```

Output of how the Compute Node was configured:

```bash
[root@Rocky-Linux-8-10-Warewulf-Controller-Node ~]# wwctl node list -a warewulf-compute-node-1
  NODE                           FIELD                                                    PROFILE  VALUE                                                  
  warewulf-compute-node-1  Id                                                       --       warewulf-compute-node-1                          
  warewulf-compute-node-1  Comment                                                  default  This profile is automatically included for each node   
  warewulf-compute-node-1  ContainerName                                            default  rockylinux-8                                           
  warewulf-compute-node-1  Ipxe                                                     --       (default)                                              
  warewulf-compute-node-1  RuntimeOverlay                                           --       (generic)                                              
  warewulf-compute-node-1  SystemOverlay                                            --       (wwinit)                                               
  warewulf-compute-node-1  Root                                                     --       (initramfs)                                            
  warewulf-compute-node-1  Discoverable                                             --       false                                                  
  warewulf-compute-node-1  Init                                                     --       (/sbin/init)                                           
  warewulf-compute-node-1  Kernel.Args                                              --       (quiet crashkernel=no vga=791 net.naming-scheme=v238)  
  warewulf-compute-node-1  Profiles                                                 --       default                                                
  warewulf-compute-node-1  PrimaryNetDev                                            --       (default)                                              
  warewulf-compute-node-1  NetDevs[default].Type                                    --       (ethernet)                                             
  warewulf-compute-node-1  NetDevs[default].OnBoot                                  --       (true)                                                 
  warewulf-compute-node-1  NetDevs[default].Hwaddr                                  --       5a:00:05:3a:6f:5a                                      
  warewulf-compute-node-1  NetDevs[default].Ipaddr                                  --       10.25.96.4                                             
  warewulf-compute-node-1  NetDevs[default].Netmask                                 default  255.255.240.0                                          
  warewulf-compute-node-1  NetDevs[default].Gateway                                 default  10.25.96.3                                             
  warewulf-compute-node-1  NetDevs[default].Primary                                 --       (true)                                                 
  warewulf-compute-node-1  Disks[/dev/vda].WipeTable                                --       true                                                   
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[boot].Number                  --       1                                                      
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[boot].SizeMiB                 --       1024                                                   
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[boot].ShouldExist             --       true                                                   
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[root].Number                  --       3                                                      
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[root].ShouldExist             --       true                                                   
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/boot].Format          --       ext4                                                   
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/boot].Path            --       /boot                                                  
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/boot].WipeFileSystem  --       true                                                   
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/root].Format          --       btrfs                                                  
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/root].Path            --       /                                                      
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/root].WipeFileSystem  --       true
```

Ran `exec` to access the Rocky Linux 8.10 container:

```bash
wwctl container exec rockylinux-8 /bin/bash
```

Set up the environment, then cloned and built the `Ignition Partition Management Module` from inside the container:

```bash
dnf groupinstall -y "Development Tools"
dnf install -y gdisk git libblkid-devel make 
wget https://go.dev/dl/go1.23.4.linux-amd64.tar.gz
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.23.4.linux-amd64.tar.gz
```

* Added the following to the end of the ~/.profile file:

```bash
cat << "EOF" | tee ~/.profile
export PATH=$PATH:/usr/local/go/bin
EOF
```

* Added `~/.profile` to `$PATH`:

```bash
source ~/.profile
```

* Installed `ignition`:

```bash
cd /tmp && git clone https://github.com/coreos/ignition.git
cd /tmp/ignition && make all && make install
```

Ran `exit` to then leave and rebuild the container.

Deployed the container to the Compute Node by restarting the node.

## Notes

The above method is not officially supported by CIQ and is considered a workaround until the `gdisk` and `ignition` packages are available in the Rocky Linux 8.x repositories.

## References & related articles

Ignition Filesystem Documentation: https://coreos.github.io/ignition/operator-notes/#filesystem-reuse-semantics/

Warewulf Disk Management: https://warewulf.org/docs/v4.5.x/contents/disks.html#disk-management/


How to Setup Ignition with Warewulf on Rocky Linux 8.x

  
## Introduction

CIQ had a customer issue, where the customer was attempting to set up local users and then authenticate using `Kerberos` to their `Active Directory` server. The following guide will explain how this was resolved.

## Problem

The customer was unable to authenticate via `ssh` to their Rocky Linux 9.2 LTS node using a local user account stored on the node. No passwords were locally saved in the `/etc/shadow` file and the passwords were only available on the `Active Directory` server.

The customer's configuration of `sssd`, `Kerberos` and `Active Directory` had worked without issue on CentOS 7.9, CentOS 8.1 and Rocky Linux 8.5. It was only on Rocky Linux 9.2 LTS where the customer was facing issues.

## Symptoms

When attempting to log in to the server via `ssh`, the following authentication messages were observed under `/var/log/secure`:

```bash
Dec 9 11:07:31 <server_name> sshd[1155]: Received signal 15; terminating.
Dec 9 11:07:31 <server_name> sshd[79752]: Server listening on 0.0.0.0 port 22.
Dec 9 11:07:31 <server_name> sshd[79752]: Server listening on :: port 22.
Dec 9 11:10:04 <server_name> unix_chkpwd[79934]: check pass; user unknown
Dec 9 11:10:04 <server_name> unix_chkpwd[79934]: password check failed for user (<username>)
Dec 9 11:10:04 <server_name> sshd[79919]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<ip_address> user=<username>
Dec 9 11:10:44 <server_name> sshd[79919]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<ip_address> user=<username>
Dec 9 11:10:44 <server_name> sshd[79919]: pam_sss(sshd:auth): received for user <username>: 4 (System error)
```

Account retrieval errors could also be seen via the `/var/log/sssd/sssd_pam.log` and `/var/log/sssd/sssd_nss.log` files:

```bash
[root@<server_name> ~]# cat /var/log/sssd/sssd_pam.log
(2024-12-16 13:45:58): [pam] [server_setup] (0x3f7c0): Starting with debug level = 0x0070
(2024-12-16 13:47:20): [pam] [cache_req_common_process_dp_reply] (0x3f7c0): [CID#1] CR #3: Could not get account info [<account_number>]: SSSD is offline

[root@<server_name> ~]# cat /var/log/sssd/sssd_nss.log
(2024-12-16 13:45:58): [nss] [server_setup] (0x3f7c0): Starting with debug level = 0x0070
(2024-12-16 13:47:20): [nss] [cache_req_common_process_dp_reply] (0x3f7c0): [CID#5] CR #17: Could not get account info [<account_number>]: SSSD is offline
```

## Resolution

The customer was utilizing `id_provider = files` in their `/etc/sssd/sssd.conf` file. Since the Rocky Linux 9.x series, [this is a deprecated option in sssd](https://sssd.io/docs/files-provider-deprecation.html).

Instead, they added the following settings into their `sssd.conf` file:

```bash
id_provider = proxy
krb5_server = example.com:88
proxy_lib_name = files
```

Then restarted the `sssd` service:

```bash
systemctl restart sssd`
```

The customer's `sssd.conf` file ultimately looked like the following (all sensitive customer information has been obfuscated):

```bash
[sssd]
    services = nss, pam
    domains = EXAMPLE.COM
    default_domain_suffix = example.com

[domain/EXAMPLE.COM]
    id_provider = proxy
    proxy_lib_name = files
    auth_provider = krb5
    krb5_realm = EXAMPLE.COM
    krb5_server = example.com:88
    krb5_validate = false
    krb5_ccachedir = /var/tmp
    krb5_keytab = /etc/krb5.keytab

[nss]
filter_groups = root,<local user>
filter_users = root,<local user>
```

In addition, the customer's `/etc/kr5b.conf` file:

```bash
includedir /etc/krb5.conf.d/

[logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

[libdefaults]
    default_realm=EXAMPLE.COM
    dns_lookup_realm = true
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    default_ccache_name = KEYRING:persistent:%{uid}

[realms]
     EXAMPLE.COM = {
     kdc = example.com:88
     admin_server = example.com:749
     }
     example.com = {
     kdc = example.com:88
     admin_server = example.com:749
     }

[domain_realm]
      example.com = EXAMPLE.COM
      example.com = EXAMPLE.COM

[appdefaults]
 pam = {
   debug = true
   ticket_lifetime = 36000
   renew_lifetime = 36000
   forwardable = true
   krb4_convert = false
 }
```

Furthermore, the customer's `/etc/nsswitch.conf` file:

```bash
# In order of likelihood of use to accelerate lookup.
shadow:     files
hosts:      files dns myhostname

aliases:    files
ethers:     files
gshadow:    files
# Allow initgroups to default to the setting for group.
# initgroups: files
networks:   files dns
protocols:  files
publickey:  files
rpc:        files
```

## Troubleshooting notes

Adding `debug_level = 9 ` to each section of the `sssd.conf` file, yields more verbose logs under `/var/log/sssd/*`

When troubleshooting the issue, the following logs are essential:

```bash
/var/log/sssd/*
/var/log/secure
```

In addition, these files are also important to check:

```bash
/etc/kr5b.conf
/etc/nsswitch.conf
/etc/pam.d/password-auth
/etc/pam.d/system-auth
/etc/sssd/sssd.conf
```

Next are these command outputs to run:

```bash
getent passwd <username>
klist
kinit <username>
ssh -o PubkeyAuthentication=no -vvv -k <username>@<server_name>
```

Finally, generating an `sosreport` from the node.

## References & related articles

[Mapping local users to Kerberos principals with SSSD](https://blog.oddbit.com/post/2015-07-16-mapping-local-users-to-kerbero/)

[sssd.conf man page](https://manpages.org/sssdconf/5)

[sssd-files man page](https://man.archlinux.org/man/sssd-files.5.en)

[sssd-krb5 man page](https://linux.die.net/man/5/sssd-krb5)


Enabling Local User Login with Active Directory Authentication using Kerberos


## Introduction

When installing Rocky Linux in a cloud environment, all directories, including `/tmp`, will typically be installed on a single volume. If you try to use a separate partition or volume for `/tmp`, you might encounter an issue where the entire root filesystem becomes read-only after a reboot. This occurs because the `systemd` `tmp.mount` unit is sometimes masked by default.

When a `systemd` unit is masked, it is linked to `/dev/null`, making it impossible to start. This differs from disabling a service, which prevents it from running at startup, but does allow it to be manually started or started by another process or unit.

You want the `tmp.mount` unit to be disabled so it does not start by default. However, you need `systemd` to start the `tmp.mount` service when it runs the `systemd-fstab-generator` at boot, which converts `/etc/fstab` entries into native `systemd` units. If `tmp.mount` is masked, this process fails, causing the system to mount the root directory as read-only.

## Symptoms

After you add an entry for `/tmp` in `/etc/fstab` and then reboot, you may see many errors related to a read-only filesystem. You will likely also see the following error when attempting to create a file:

```bash
touch /var/tmp/test
touch: cannot touch 'var/tmp/test': Read-only file system
```

## Resolution

Telling `systemd` to unmask this unit will allow you to mount `/tmp` according to how `/etc/fstab` is configured:

```bash
sudo systemctl unmask tmp.mount
```

If you get a read-only filesystem error when attempting this, you need to remount the root volume first:

```bash
sudo mount -o remount,rw /
```

Once the filesystem is remounted and you have verified you can write to it, rerun the `unmask` command. After a reboot, your server should start up as expected.

There is no need to enable this unit as `systemd` will dynamically generate it and start it at boot.

## Root Cause

This issue arises because cloud servers are primarily used for network-specific tasks such as web hosting. 

As a result, the need for an in-memory `/tmp` directory (which is generated by default when you start `tmp.mount`), is largely negated because the network is significantly slower than the disk that `/tmp` would run on. 

Additionally, since memory is a significant factor in the cost of cloud services, Rocky Linux does not rely on memory-based `tmpfs` directories. Moreover, the need for a separate partition or volume for `/tmp` is only necessary in minimal use cases and is unnecessary for the majority of workloads.

## References & related articles

freedesktop.org's page on `systemd-fstab-generator`: [Link](https://www.freedesktop.org/software/systemd/man/latest/systemd-fstab-generator.html)
Rocky Linux Admin Guide [Link](https://rocky-linux.github.io/documentation/RockyLinuxAdminGuide.pdf)

How to Avoid a Read-only Filesystem When Using a Separate Partition or Volume for /tmp


## Introduction

The key to this problem is *entropy*. The FIPS openssl version is built with strict requirements for entropy to provide the necessary security guarantees: it fills up "pools" of truly random numbers when it starts up. If these pools aren't filled, then applications that call the openssl library will block until it can gather enough random material to proceed. Regular openssl doesn't do this: if it can't get enough true randomness, it uses fuzzy pseudorandomness and continues. It is important to understand that FIPS is not meant to be run in pieces. It is a holistic system. The kernel provides the entropy to the openssl library via a system call.

The RHEL/Rocky 9 kernel (and upstream 5.10, 5.15, etc. kernels) seem to provide enough entropy more or less built-in. CentOS 7 (kernel 3.10.0) and RHEL/Rocky 8 (kernel 4.18) do not in some cases.

## Problem

When using openssl fips package on a CentOS/Rocky 8 system you may see the following timeouts when using dnf with the FIPS openssl libraries installed:

```
dnf list
Rocky Linux BaseOS         0.0  B/s |   0  B     17:45
Errors during downloading metadata for repository 'baseos':
 - Curl error (28): Timeout was reached for https://.../BaseOS/x86_64/os/repodata/repomd.xml [SSL connection timeout]
 - Curl error (28): Timeout was reached for https://.../BaseOS/x86_64/os/repodata/repomd.xml [Operation timed out after 99165 milliseconds with 0 out of 0 bytes received]
Error: Failed to download metadata for repo 'baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
```

Also, sshd may freeze on startup, preventing remote access to a system.

## Resolution

Fortunately, the solution is simple: On any CentOS 7/Rocky 8 host, if you want to run containers with the FIPS openssl library installed, you need to do this (on the host, not inside the Rocky 8 container):

```
yum install rng-tools

systemctl enable --now rngd
```

The rngd daemon feeds extra, hardware generated, high-quality sources of entropy into the kernel, thus preventing the entropy pool from becoming exhausted.


Configuring Entropy on CentOS7/RockyLinux8 for FIPS OpenSSL


## Introduction

Fuzzball Orchestrate is architected as a collection of microservices deployed in a Kubernetes (K8s)
cluster. To understand what is happening with the Fuzzball deployment, you must be able to inspect
the status of the pods supporting these containerized microservices, and gather relevant logs.

## Checking pod status

Any Fuzzball troubleshooting starts with confirming the presence and inspecting the status of
containers/microservices within their respective pods.

You can view and inspect the status of pods hosting containerized microservices using the standard
K8s control plane tool, `kubectl`. After executing the following `kubectl` command, a table
is displayed giving information about all of the pods within the K8s cluster.

```text
# # kubectl get pods -A
NAMESPACE            NAME                                                       READY   STATUS      RESTARTS      AGE
cert-manager         cert-manager-56f459fcb7-j7wmt                              1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-56f459fcb7-m8gq2                              1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-56f459fcb7-qj9zd                              1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-cainjector-78f497d9d8-5fc5r                   1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-webhook-5bf5cccd59-qg5qs                      1/1     Running     1 (35m ago)   19d
cert-manager         trust-manager-778f7b488-gdmrh                              1/1     Running     1 (35m ago)   19d
database             database-debug-0                                           1/1     Running     1 (35m ago)   19d
database             database-postgresql-0                                      1/1     Running     1 (35m ago)   19d
fuzzball-system      fuzzball-operator-controller-manager-55cc86985f-c7tzn      2/2     Running     2 (35m ago)   19d
fuzzball             audit-archive-28836720-4wcf5                               0/1     Completed   0             2d6h
fuzzball             audit-archive-28838160-86zjk                               0/1     Completed   0             30h
fuzzball             audit-archive-28839600-tbhj2                               0/1     Completed   0             6h25m
fuzzball             fuzzball-account-7749678b7c-m272g                          1/1     Running     1 (35m ago)   19d
fuzzball             fuzzball-account-external-5f5cd6f56-86k4z                  1/1     Running     1 (35m ago)   19d
fuzzball             fuzzball-account-worker-7f69c98747-gbtvc                   1/1     Running     4 (33m ago)   19d
fuzzball             fuzzball-admin-0                                           1/1     Running     1 (35m ago)   19d
[...snip]
```

If you want to view only pods running within the `fuzzball` namespace, you can change the command
like so:

```text
# kubectl get pods -n fuzzball 
NAME                                             READY   STATUS      RESTARTS       AGE
audit-archive-28836720-4wcf5                     0/1     Completed   0              2d8h
audit-archive-28838160-86zjk                     0/1     Completed   0              32h
audit-archive-28839600-tbhj2                     0/1     Completed   0              8h
fuzzball-account-77bc99544-n2dgb                 1/1     Running     0              119s
fuzzball-account-external-7977f8c5dc-hmfhf       1/1     Running     0              119s
fuzzball-account-worker-6b646c8b76-jq2sf         1/1     Running     0              119s
fuzzball-admin-0                                 1/1     Running     1 (173m ago)   19d
fuzzball-audit-74bb656b6b-lltx4                  1/1     Running     0              119s
fuzzball-auth-spicedb-57bb79cf9b-k2zh7           1/1     Running     0              119s
fuzzball-dns-569f675947-cf59z                    1/1     Running     0              119s
fuzzball-kube-68fc8d8b8d-f7gkg                   1/1     Running     0              119s
fuzzball-log-845fc88dd9-bvr4c                    1/1     Running     0              119s
fuzzball-openapi-686488649-7fj2s                 1/1     Running     0              119s
fuzzball-organization-7f7f5f6878-qs55p           1/1     Running     0              119s
fuzzball-organization-external-cc78884d5-z56j5   1/1     Running     0              118s
fuzzball-organization-worker-55599cd8d8-htknd    1/1     Running     0              118s
fuzzball-permission-external-55d965c957-n5fsv    1/1     Running     0              118s
fuzzball-schedule-0                              1/1     Running     0              170m
[...snip]
```

Note the values in the `READY` column indicating `1/1`, (or potentially `2/2`, etc.) and the
`STATUS` column indicating `Running`. Any pods that do not report a `Running` or `Completed` status,
or those that report fewer instances than expected should be investigated further by inspecting
logs.

## Retrieving and Inspecting logs

While provisioning Fuzzball within a K8s cluster (using `kubectl apply -f fuzzball.yaml`), you can
watch the log output with the following command:

```text
# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1
```

You can stream logs from an individual pod using the following syntax. Note that the namespace must
be specified since Fuzzball pods are not deployed in the "default" namespace. For example, to view
the logs streaming from the `fuzzball-schedule-0` pod, you could use the following command. (Omit
the `--follow --tail=0` options/arguments if you want to view the log at a single instant without
streaming it.)

```text
# kubectl logs -n fuzzball fuzzball-admin-0 --follow --tail=0
```

The `kubectl logs` command outputs verbose logs in json format. These can be filtered and
parsed into human-readable output using `jq` like so:

```text
# kubectl logs -n fuzzball fuzzball-admin-0 | \
    jq -r 'select(.level != "debug") | [.ts, .msg, .Error] | @tsv'
```

The `kubectl` utility does not provide a straightforward command for capturing all Fuzzball logs in
a single stream. However, the logs for a collected Fuzzball “app” can be inspected using a label
selector like so. (This is the strategy used above to stream logs during the provisioning process.)

```text
# kubectl logs --prefix --namespace=fuzzball --all-containers \
    --max-log-requests=99 --tail=0 --follow \
    --selector='app.kubernetes.io/name=fuzzball-schedule'
```

The list of available labels can be viewed using `kubectl get pods -n fuzzball --show-labels`.

Finally, it may be useful to gather a full log dump (for example, to send to CIQ support). This can
be accomplished by requesting logs from all app labels. The app labels may change with different
versions of Fuzzball and can be inspected as described above.  Here is an example:

```text
# kubectl logs --prefix --all-containers --max-log-requests=99 \
    --namespace=fuzzball \
    --selector='app.kubernetes.io/name in (fuzzball-account, fuzzball-admin, fuzzball-audit, fuzzball-auth, fuzzball-dns, fuzzball-kube, fuzzball-log, fuzzball-openapi, fuzzball-organization, fuzzball-permission, fuzzball-schedule, fuzzball-secret, fuzzball-storage, fuzzball-sync, fuzzball-ui, fuzzball-user, fuzzball-workflow)'
```

## Restarting Fuzzball

You can invoke the command `kubectl rollout restart` to restart the Fuzzball deployment. For
example:

```text
# kubectl rollout restart deployment -n fuzzball 
```

Other namespaces that may be pertinent for restart include

- `database` - provides the PostgreSQL database that backs Fuzzball Orchestrate state
- `ingress` - maps HTTP(S) URLs to relevant Fuzzball services, including the Fuzzball UI, Fuzzball API, and Keycloak
- `kyverno` - provides a policy engine used by Fuzzball Orchestrate
- `workflow` - provides the temporal service that is used by Fuzzball Orchestrate
- `identity` - provides a Keycloak instance for identity and access management

## Uninstalling Fuzzball

If a deployment failed or you have a need to uninstall a deployed Fuzzball installation, you may
need to tear down the installation and start fresh. The following command will delete an existing
Fuzzball deployment, allowing you to start over if needed.

```text
# kubectl delete FuzzballOrchestrate fuzzball-orchestrate
```

You can watch the deprovisioning process as it occurs with the following command:

```text
# kubectl logs -n fuzzball-system -l app.kubernetes.io/name=fuzzball-operator -f --tail=-1
```

You may also want to remove the `substrate` configuration directory that is shared out to compute
nodes via NFS.  For instance:

```text
# # rm -rf /srv/fuzzball/shared/substrate
```

This directory will be recreated if you re-provision your Fuzzball instance.


Basic Troubleshooting Procedures


## Introduction

The need for secure and efficient file transfer solutions has become paramount. Whether it is sharing sensitive data, collaborating on projects, or automating workflows, a reliable Secure File Transfer Protocol (SFTP) server can streamline these processes while ensuring data integrity and confidentiality.

The CIQ Secure File Exchange (SFE) service is an SFTP, FTP/S, HTTP/S, and WebDAV server. It provides a flexible foundation for managing file transfers while offering an efficient and scalable solution for storing and retrieving data. The service supports private virtual folders per user, which can also be shared with other users as well.

## How to access the service

To have an SFE account generated, please create a ticket via the CIQ Support Portal. A member of support will then generate your account and securely send you the login credentials via the ticket.

Login to the [CIQ SFE Client Portal](https://sftp.ciq.com/web/client/login) using your provided credentials.

You will then be prompted to change your password. Please do so and store the new password in a secure location.

## How to upload files

Files can be uploaded in two ways: via your web browser or the CLI.

### Web browser method

Log in with your credentials at the [CIQ SFE Client Portal](https://sftp.ciq.com/web/client/login).

Navigate to the `My Files` section and press the `Upload files` button.

![An image showing the CIQ SFE Client's My Files section and highlighting the "Upload" button that can be clicked to open the file upload dialogue box.](/images/articles/ciq_sfe_file_upload_1.webp)

Select your files and then hit `Submit`:

![An image from the CIQ SFE Client, highlighting the area where the user can upload their files and then clicking the "Submit" button to start the upload.](/images/articles/ciq_sfe_file_upload_2.webp)

The file will be uploaded to your virtual folder. Please inform the Support Team via the relevant support ticket and they can retrieve the files and start their investigation.

**If you would like to further customize how you share your files, please follow the steps below:**

Under `My Files`, click the checkboxes next to the files you uploaded and then click the `Share` button:

![An image from the My Files section of the CIQ SFE Client, with the files that the user wants to share having their checkboxes ticked.](/images/articles/ciq_sfe_file_upload_4.webp)

![An image from the My Files part of the CIQ SFE Client, highlighting the Share button to click, so that the user can share their files.](/images/articles/ciq_sfe_file_upload_3.webp)

At the next screen, add a `Name` for your share, set a `Password` and `Expiration` date if you desire and then select `Submit`:

![An image of the "Add a new share" section of the CIQ SFE Client, showing a "Name" can be added for the share, the "Scope" can be changed from "Read" to "Write" to "Read and Write", a Password can be added, as well an Expiration date. Finally the "Submit" button is highlighted in green for the user to click and the share can be created.](/images/articles/ciq_sfe_file_upload_5.webp)

Then in the `Shares` screen, select the share that you created and click the `Link` button at the top of the page:

![An image showing the test share that was created in the previous step and has been selected for sharing by the user.](/images/articles/ciq_sfe_file_upload_7.webp)

![An image displaying the "Shares" section of the CIQ SFE Client, showing the "Link" button where links to the share can be generated.](/images/articles/ciq_sfe_file_upload_6.webp)

Finally, copy the link type you would like to share and include the link in the relevant ticket. The Support Team can then easily access your files:

![An image showing that once the "Link" button is clicked, three links are generated: 1) The user can download the share as a zip file. 2) The user can access a directory to view all of the files. 3) If only a single file is shared, the user can download the file uncompressed.](/images/articles/ciq_sfe_file_upload_8.webp)

### CLI method

Run the following command to access the CIQ SFE service via the CLI:

```bash
sftp -P 2022 <YOUR_USERNAME>@ciq.com@sftp.ciq.com
```

Enter your password when prompted.

Create a directory where you would like to store your files:

```bash
sftp> mkdir <YOUR_DIRECTORY>
```

Upload your files using the `put` command:

```bash
put ./text.txt ./test/
```

An example is below of the author uploading a file called `test.txt` to a directory in the CIQ SFE service called `test`:

```bash
sftp> put ./test.txt test/
Uploading ./test.txt to /test/test.txt
test.txt                                                                                                                                      100%   13     0.1KB/s   00:00    
sftp> 
```

Once uploaded, please check that the files were uploaded successfully. An example is shown below:

```bash
sftp> ls -l test/ 
-rwxr-xr-x    1 991      991            13 Jan 20 02:32 test.txt
```

Please inform the Support Team via your ticket that the files have been uploaded successfully.

## Important!

For all methods listed above, **please provide a sha256sum in the ticket for each file that is in the multiple gigabytes**. This allows the Support Team to easily verify if an upload has been successful or not and can continue their investigation without further interruption.

If you run into any problems whilst uploading files to the CIQ SFE service, please don't hesitate to make the Support Team aware and they will assist you with the uploading process.

How to Upload Files to the CIQ Secure File Exchange (SFE) Service


## Introduction

Using PXE to bootstrap nodes with an image is commonplace in many business environments and has made both deploying and managing an image across a fleet of nodes at scale trivial.

This article will go through the setup process of deploying a PXE server on Rocky Linux 9.5 and then bootstrap a fresh node with a Rocky Linux 9.5 image. The article will focus on a UEFI boot use case, as opposed to a legacy BIOS boot.

## Prerequisites

* A node with Rocky Linux 9.5 installed on it. This will be the PXE server.

* A node that has PXE support available. This will be the PXE client.

* A Rocky Linux 9.5 DVD ISO.

## Instructions

### PXE Server Setup

To set up the PXE Server please follow the steps below:

* Update all packages:

```
sudo dnf update -y
```

* Install `dnsmasq` to provide a DNS and DHCP server support:

```bash
sudo dnf install -y dnsmasq
```

* Set up your `dnsmasq.conf` configuration with the appropriate subnet, netmask, broadcast-address, and more:

```bash
cat << "EOF" | sudo tee /etc/dnsmasq.conf
interface=<your_ethernet_interface>,lo
#bind-interfaces
domain=<name_here-anything_is_okay>
# DHCP range-leases
dhcp-range=<your_ethernet_interface>,<start_of_dhcp_range>,<end_of_dhcp_range>,255.255.240.0,1h
# PXE
dhcp-boot=pxelinux.0,PXEserver,<your_rocky_linux_node_address>
# Gateway
dhcp-option=3,<your_gateway_address>
# DNS
dhcp-option=6,<your_dns_server_address>
server=<your_rocky_linux_node_address>
# Broadcast Address
dhcp-option=28,<your_broadcast_address>
# NTP Server
dhcp-option=42,0.0.0.0

PXE-prompt="Press F8 for menu.", 60
PXE-service=x86PC, "Install Rocky Linux 9.5 from your PXE server <your_rocky_linux_node_address>", pxelinux
enable-tftp
tftp-root=/var/lib/tftpboot
EOF
```

Example:
*Please supplement the addresses with those from your own environment*

```bash
cat << "EOF" | sudo tee /etc/dnsmasq.conf
interface=enp8s0,lo
#bind-interfaces
domain=test
# DHCP range-leases
dhcp-range=enp8s0,10.25.96.4,10.25.96.5,255.255.240.0,1h
# PXE
dhcp-boot=pxelinux.0,pxeserver,10.25.96.3
# Gateway
dhcp-option=3,10.25.96.1
# DNS
dhcp-option=6,10.25.96.1
server=10.25.96.3
# Broadcast Address
dhcp-option=28,10.25.96.255
# NTP Server
dhcp-option=42,0.0.0.0

pxe-prompt="Press F8 for menu.", 60
pxe-service=x86PC, "Install Rocky 9.5 from network server 10.25.96.3", pxelinux
enable-tftp
tftp-root=/var/lib/tftpboot
EOF
```

*The PXE server has been assigned the address of `10.25.96.3` in the above configuration.*

* Install the `syslinux` package:

```bash
sudo dnf install -y syslinux
```

* Install the `tftp-server` package:

```bash
sudo dnf install -y tftp-server
```

* Copy the contents of the `syslinux` directory to `/var/lib/tftpboot`:

```bash
sudo cp -r /usr/share/syslinux/* /var/lib/tftpboot
```

* Create a directory called `pxelinux.cfg`:

```bash
sudo mkdir /var/lib/tftpboot/pxelinux.cfg
```

* Generate the PXE BOOT menu:

```bash
cat << "EOF" | sudo tee /var/lib/tftpboot/pxelinux.cfg/default
default menu.c32
prompt 0
timeout 300
ONTIMEOUT local

menu title # Rocky Linux 9.5 Boot Menu #

label 1
menu label ^1) Install Rocky Linux 9.5 x64 using a Remote Repository
  kernel rocky95/vmlinuz
  append initrd=rocky95/initrd.img inst.zram=1 inst.repo=https://dl.rockylinux.org/pub/rocky/9.5/BaseOS/x86_64/os/
EOF
```

**NOTE**
The PXE BOOT menu can be further expanded to include `kickstart` file configurations and other customizations. An example entry using a `kickstart` file is below:

```bash
label 2
menu label ^2) Install Rocky Linux 9.5 x64 using a Remote Repository
  kernel rocky95/vmlinuz 
  append initrd=rocky95/initrd.img inst.debug ip=dhcp inst.ks=nfs:<your_rocky_linux_node_ip>:</path/to/your/kickstarter_cfg_file> inst.repo=nfs:<your_rocky_linux_node_ip>:</path/to/your/iso> inst.zram=1
```

* Create a directory called `rocky95`:

```bash
sudo mkdir /var/lib/tftpboot/rocky95
```

* Mount the Rocky Linux 9.5 DVD ISO image:

```bash
sudo mount -o loop </path/to/iso> </iso/mount_point>
```

* For this example we mounted the Rocky Linux 9.5 ISO at `/mnt`:

```bash
sudo mount -o loop ~/isos/Rocky-9.5-x86_64-dvd.iso /mnt
```

* Copy the required `initrd.img` and `vmlinuz` files to `/var/lib/tftpboot/rocky95`:

```bash
sudo cp /mnt/images/pxeboot/vmlinuz /var/lib/tftpboot/rocky95
sudo cp /mnt/images/pxeboot/initrd.img /var/lib/tftpboot/rocky95
```

* Start and enable the `dnsmasq` and `tftp` services at boot:

```bash
sudo systemctl enable --now dnsmasq
sudo systemctl enable --now tftp
```

* Allow `dhcp`, `dns`, `proxydhcp` and `tftp` traffic through your firewall:

```bash
sudo firewall-cmd --add-service=dhcp --permanent
sudo firewall-cmd --add-service=dns --permanent
sudo firewall-cmd --add-port=69/udp --permanent
sudo firewall-cmd --add-port=4011/udp --permanent
sudo firewall-cmd --reload
```

### PXE Client Setup

To set up the PXE Client please follow the steps below to enable the available PXE option from the client machine's BIOS settings. In this example, the following needed to be enabled:

* Navigate to `Advanced` --> `Onboard Devices Configuration` --> `Realtek LAN Controller` and set this option to `Enabled`.

* Navigate to `Advanced` --> `Onboard Devices Configuration` --> `Realtek LAN Controller` --> `Realtek PXE Option ROM` and configured this option as `Enabled`.

* Navigate to `Advanced` --> `Network Stack Configuration` --> `Network Stack` and also set this option as `Enabled`.

* Finally went to `Advanced` --> `Network Stack Configuration` --> `Ipv4 PXE Support` and changed this to `Enabled`.

* Restart your PXE client machine and it should connect to your PXE server. 

* Press `F8` at the prompt:

![The prompt to press F8 shows that the PXE client machine has connected to the PXE server successfully. Press F8 to continue.](/images/articles/pxe_server_boot_1.webp)

* Press the `Enter` key at the next screen and proceed to the Rocky Linux 9.5 Boot Menu. Select `option 1`:

![The Rocky Linux 9.5 PXE Boot Menu appears. One option called "Install Rocky Linux 9.5 x64 using a Remote Repository" is present. The user is prompted to select that option.](/images/articles/pxe_server_boot_2.webp)

Your PXE client will now boot up and download the `initrd.img` and `vmlinuz` files, as well as the files from the remote repository. The Anaconda installer will then appear and you can continue the installation as normal. Of course, if you submit a `kickstart` configuration file, the installation part of the process can be skipped.

## Conclusion

This guide was built in the hope of helping bring up a simple PXE server. It touched on configurations for `dnsmasq`, `tftp`, setting up the PXE server boot menu, and more. PXE is an amazing technology and has truly revolutionized large scale deployments.

How to Setup a PXE Server on Rocky Linux 9.x


## Introduction

ICMP (Internet Control Message Protocol) is a fundamental part of networking, commonly used for diagnostic tools like `ping`. It is a set of rules used by devices to report data transmission errors within a network. During message exchanges between a sender and receiver, unexpected errors may occur, such as messages being too long or data packets arriving out of order, preventing proper assembly. In these instances, the receiver uses ICMP to notify the sender of the error and requests the message to be resent.

## Problem

Limiting the rate of ICMP echo replies is an important security measure to mitigate the risk of ICMP-based attacks, such as ICMP flood attacks. While the `tc` (Traffic Control) command is commonly used to manage traffic flow on Linux, including rate-limiting ICMP replies, an alternative approach is to use iptables through Firewalld's [direct](https://firewalld.org/documentation/direct/) rules.

## Resolution

### Create a new rule

The appropriate limits depend on your security policy and environment, but a good starting point is a rate of 1 reply per second with a burst limit of 5. This allows the server to respond to up to 5 pings in quick succession before enforcing the 1-per-second restriction. The burst setting also helps accommodate short spikes in legitimate traffic:

```shell
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 10 -p icmp --icmp-type echo-request -j DROP
firewall-cmd --reload
```

The `0` and `10` in these rules specify their priority within firewalld. Firewalld processes rules in ascending order of priority, so the first rule with priority 0 will be evaluated before the second rule with priority 10. If traffic matches the first rule meaning its within the rated limit, it will be accepted and will not proceed to the second rule. If the traffic does not match the first rule meaning it exceeds the rate limit, it will advance to the second rule and will be dropped.

### Replace the existing rule

If you need to modify this rule, you will need to delete the rule and recreate it. To see your current list of rules:

```shell
[root@rockylinux ~]# firewall-cmd --direct --get-all-rules

ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
ipv4 filter INPUT 10 -p icmp --icmp-type echo-request -j DROP
```

Now, append the rule you want to delete to `firewall-cmd --direct --permanent --remove-rule`. When adding a new rule, ensure that your priority is set lower than the priority of the `DROP` rule. In our case we will keep with the same priority of `0`:

```shell
firewall-cmd --direct --permanent --remove-rule ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 2/sec --limit-burst 7 -j ACCEPT
firewall-cmd --reload
```

### Delete the rules

To remove these rules completely, we can use the same `--remove-rule` flag as mentioned above. First start by viewing the existing list of rules as before:

```shell
[root@rockylinux ~]# firewall-cmd --direct --get-all-rules

ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
ipv4 filter INPUT 10 -p icmp --icmp-type echo-request -j DROP
```

Now, append the rules you want to delete to `firewall-cmd --direct --permanent --remove-rule`:

```shell
firewall-cmd --direct --permanent --remove-rule ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
firewall-cmd --direct --permanent --remove-rule ipv4 filter INPUT 10 -p icmp --icmp-type echo-request -j DROP
firewall-cmd --reload
```

## Notes

If you need to create, modify, or delete these rules in an environment where the Firewalld service cannot be started (such as a chrooted environment or in a kickstart), you can use [firewall-offline-cmd](https://firewalld.org/documentation/man-pages/firewall-offline-cmd.html) to modify them.

## References & related articles

[Rocky Linux Beginners Guide to Firewalld](https://docs.rockylinux.org/guides/security/firewalld-beginners/)  
[Firewalld Official Documentation](https://firewalld.org/documentation/)  
[Firewalld Direct Rules Documentation](https://firewalld.org/documentation/direct/)  
[Firewalld Offline Command Documentation](https://firewalld.org/documentation/man-pages/firewall-offline-cmd.html)


Configuring ICMP Rate Limiting in Firewalld


## Introduction

It is inevitable that hardware will fail and there are hundreds of possible scenarios.

This guide will detail troubleshooting steps which can be used to identify hardware errors and how to rectify them.

## Problem

A customer informed CIQ that a node in their production cluster had spontaneously crashed and then rebooted itself.

## Symptoms

Checking `/var/log/messages` from the `sosreport`, there were hundreds of `[Hardware Error]: Hardware error from APEI Generic Hardware Error Source` messages observed.

## Resolution

### Prerequisites

Access to the `root` user or a user with escalated privileges.

### Creating an sosreport

When creating a ticket with CIQ, generating and attaching an `sosreport` to the ticket is very helpful. This allows the support team to start their investigation, without further having to ask the customer to generate one.

Install the `sos` package:

```bash
dnf install -y sos
```

Run the `sos report --batch` command to perform unattended data gather of your system.

Once complete, you will find the `sosreport` under `/var/tmp/sosreport-hostname-yyyy-mm-dd-abcdefg.tar.xz`. An example is below:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# ls -l /var/tmp/ | grep sosreport
-rw-------. 1 root root 11094320 Dec 27 02:33 sosreport-Rocky-Linux-9-5-Test-Machine-2024-12-27-elpmjjq.tar.xz
-rw-r--r--. 1 root root       65 Dec 27 02:33 sosreport-Rocky-Linux-9-5-Test-Machine-2024-12-27-elpmjjq.tar.xz.sha256
```

Please upload the `sosreport` to the ticket.

### Analysis of the issue

When observing behavior such as crashes or random reboots, the best log source is `/var/log/messages` - this records the global messages that are generated from the various services running on your server.

Hardware errors are usually identified by the kernel in the `/var/log/messages` log with a `[Hardware Error]` prefix. These messages come from the ACPI Platform Error Interface.

Using a combination of tools such as `grep`, `awk`, and `uniq` you can quickly find and triage repeating errors. The following in `/var/log/messages` from a customer's `sosreport` shows hundreds of memory-related errors:

```bash
grep "Hardware Error" ./sosreport-<NODE_NAME>-2024-12-24-gwwbeqs/var/log/messages | awk '{print $8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20}' | grep "memory error" | uniq -c
 710 section_type: memory error
```

In addition to when a `Hardware Error` occurs, you will also encounter messages of `Corrected error, no action required`. While the error may be temporarily corrected, if these messages continue, it is a sign of a more serious fault. In that scenario, it is best to contact the hardware vendor and replace the affected part outright.

### Recommendations

In the above memory fault example, the recommendation to the customer was to change out their memory modules and they then contacted their hardware team for further analysis.

## References & related articles

`awk` man page: https://man7.org/linux/man-pages/man1/awk.1p.html

`grep` man page: https://man7.org/linux/man-pages/man1/grep.1.html

`sosreport` man page: https://linux.die.net/man/1/sosreport

`uniq` man page: https://man7.org/linux/man-pages/man1/uniq.1.html


How to Identify Hardware Faults in Rocky Linux


## Introduction
A quick intro to the topic at hand.

In some technical processes, reading from and writing to registers may lead to inconsistencies or invalid values due to the simultaneous execution of read and write operations. This article provides an overview of this issue and offers potential solutions for troubleshooting it. 

## Problem
If gathersing the pci information from /sys (sysfs) the data shows pci link speed and status. In some cases, the LnkSta that is returned indicates the following "16GT/s(downgraded)  x4(downgraded)". While 16GT/s and x4 might be the correct speed and width, the ouput shows "downgraded".

```
$ lspci | grep downgr
...
        LnkSta: Speed 8GT/s (downgraded), Width x8 (ok)
        LnkSta: Speed 16GT/s (ok), Width x8 (downgraded)
...
```
### This could match the following criteria

- Inconsistencies in data when comparing multiple reads from the same register
- Observing incorrect or unexpected values that do not align with the expected behavior.
- Frequent occurrences of invalid values (e.g., once or twice a week).

## Resolution

While this is not a resolution to the problem, this could help gathering information that you can bring to your hardware provider.

Below is the lspci code that emits the string: “(downgraded)”. Just so we know what criteria is being used here:

```
static char *link_compare(int type, int sta, int cap)
{
  if (sta > cap)
    return " (overdriven)";
  if (sta == cap)
    return "";
  if ((type == PCI_EXP_TYPE_ROOT_PORT) || (type == PCI_EXP_TYPE_DOWNSTREAM) ||
      (type == PCI_EXP_TYPE_PCIE_BRIDGE))
    return "";
  return " (downgraded)";
}
```
This is called from: cap_express_link()

```
  w = get_conf_word(d, where + PCI_EXP_LNKSTA);
  sta_speed = w & PCI_EXP_LNKSTA_SPEED;
  sta_width = (w & PCI_EXP_LNKSTA_WIDTH) >> 4;
  printf("\t\tLnkSta:\tSpeed %s%s, Width x%d%s\n",
        link_speed(sta_speed),
        link_compare(type, sta_speed, cap_speed),
        sta_width,
        link_compare(type, sta_width, cap_width));
```
The link_compare() code came from [this discussion:](https://lore.kernel.org/all/20210318171122.GA151712@bjorn-Precision-5520/T/)

In sum, it’s checking the status speed from `PCI_EXP_LNKSTA_SPEED` and if it’s less than cap_speed then it’s always reporting “(downgraded)” for devices other than `PCI_EXP_TYPE_ROOT_PORT`, `PCI_EXP_TYPE_DOWNSTREAM`, `PCI_EXP_TYPE_PCIE_BRIDGE`.

The PCI configuration data, including speed and width information that lspci code reads and potentially reports as “downgraded,” is pulled from the [sysfs](https://docs.kernel.org/PCI/sysfs-pci.html) entries under `/sys/bus/pci/devices`. As an example, for NVMe devices, the data is accessed from the corresponding entries for each NVMe device under `/sys/bus/pci/devices/[DEVICE_ID]/config`.
 
The reading of PCI configuration data is managed by the kernel's PCI subsystem, specifically within the PCI sysfs interface and related functions. The primary functions involved include:
```
pci_read_config() in drivers/pci/pci-sysfs.c
pci_user_read_config_##size in drivers/pci/access.c
```
These functions handle the reading of configuration data directly from the hardware registers of the PCI devices. 

If you use a script that reads these data, is good to understad that it is just getting raw data out of the device on the PCI bus.
 
For debugging you could try using dmesg and other system logs to check for any hardware events or errors reported by the kernel that might indicate link renegotiation or other PCIe issues. 

Look for AER (Advanced Error Reporting) messages which can sometimes provide clues about PCIe link issues:
```
$ dmesg | grep AER
```

## Possible Causes

1. Simultaneous read and write operations on the same register: This can result in capturing an intermediate state instead of the final value, leading to inconsistencies.
2. Incorrect timing between read and write operations: If there's not enough time between reading and writing, it may lead to the capture of an incorrect or invalid value.
3. Hardware malfunctions: Faulty hardware components or software glitches can cause unexpected behavior in register values.


## References & related articles 
[lspci: Don't report PCIe link downgrades for downstream ports](https://lore.kernel.org/all/20210318171122.GA151712@bjorn-Precision-5520/T/)
[Accessing PCI device resources through sysfs](https://docs.kernel.org/PCI/sysfs-pci.html)


Inconsistent Values When Reading and Writing PCI Registers


## Introduction

There are multiple ways to create an image for Warewulf that includes the NVIDIA drivers. This article will explore how to work with a Containerfile or modify a regular image.

## Prerequisites

This guide assumes that Warewulf server has been installed and configured, and at least one node has been deployed.

## Instructions

### Create an image using a Containerfile

Warewulf supports creating images directly from containers, enabling you to build a custom container that includes the required NVIDIA drivers. The [Warewulf-images](https://github.com/warewulf/warewulf-node-images/blob/main/examples/rockylinux-9-nvidia/Containerfile) GitHub repository has an example Containerfile that we can work from. Start by installing Podman on the server you want to build the image on. To keep things simple, we will install this on our Warewulf host server to streamline the import process:

```shell
sudo dnf install -y podman
```

Next, we'll create a folder to add our Containerfile to:

```shell
mkdir rockylinux-9-nvidia
cd rockylinux-9-nvidia
```

Now create a new file called `Containerfile` and add the following to it:

```Dockerfile
FROM ghcr.io/warewulf/warewulf-rockylinux:9

RUN dnf -y install dnf-plugins-core epel-release kernel-headers \
    && dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/$(arch)/cuda-rhel9.repo \
    && dnf -y module install nvidia-driver:latest-dkms \
    && dnf -y install datacenter-gpu-manager \
    && dnf clean all \
    && for dir in /usr/src/kernels/*; do dkms autoinstall --kernelver $(basename $dir); done \
    && dkms status
```

Feel free to modify this as needed to suit your environment. Next, we can build this image:

```shell
podman build -t rockylinux-9-nvidia:v1 .
```

Feel free to replace `v1` with a tag or versioning system that better fits your personal preference or corporate policy if needed. If you see an error `cannot apply additional memory protection after relocation: Permission denied`, you may need to rerun with SELinux disabled or configured:

```shell
podman build --security-opt label=disable -t rockylinux-9-nvidia:v1 .
```

And then rerun the build from before. Once the build finishes, we can save this to a `tar` file and then import it into Warewulf:

```shell
podman save -o rockylinux-9-nvidia-v1.tar localhost/rockylinux-9-nvidia:v1
wwctl image import file://rockylinux-9-nvidia-v1.tar rockylinux-9-nvidia-v1
```

Once the import finishes, you can now assign this image to a [profile or node](https://warewulf.org/docs/v4.6.x/nodes/profiles.html) as you would any other image. You can find an example of this in the following section within this guide. Another benefit of using Containerfiles, is their ability to be integrated into a CI/CD pipeline to enable version control and automated building.

### Modifying a regular image

Warewulf includes the ability to enter into an image via a shell. This allows you to modify the container image and install components like NVIDIA drivers easily and quickly. We first need to start by either copying an existing container or downloading a new container. You can use `wwctl container list` to view your current images if you would like to build off an existing container. You can, of course, edit any existing container; however, we always recommend working from a copy or creating a backup first.

In this example, we are going to start by downloading a fresh copy of Rocky Linux 9. We can do so by downloading an image from the Warewulf repository:

```shell
wwctl container import docker://ghcr.io/warewulf/warewulf-rockylinux:9 rockylinux-nvidia-9
```

In this example, the name of our container will be `rockylinux-nvidia-9`. Feel free to change this to match your environment or to add additional information such as the date or version. Once the container is finished downloading, we can enter into this container's shell and install the NVIDIA drivers:

```shell
wwctl container shell rockylinux-nvidia-9
```

You can find more detailed instructions for installing NVIDIA drivers [here](https://docs.nvidia.com/cuda/pdf/CUDA_Installation_Guide_Linux.pdf). The following are example steps for Rocky Linux 9:

```shell
# Install necessary packages and add the NVIDIA repository
dnf -y install dnf-plugins-core epel-release kernel-headers
dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/$(arch)/cuda-rhel9.repo
# Install the latest NVIDIA driver
dnf -y module install nvidia-driver:latest-dkms
```

We can verify this installed correctly by checking for the dynamically loaded kernel:

```shell
dkms status | grep nvidia
nvidia/555.42.02, 5.14.0-427.20.1.el9_4.x86_64, x86_64: installed
```

Once you have finished installing the driver and any other applications you need, type `exit` to leave the container. Warewulf should rebuild the container once you are finished, however you can run the following command manually to verify the container was built:

```shell
wwctl container build
```

Running this command will not return anything if the image has already been built/updated. If applicable, we can sync local users into the container. You can read more about the `syncuser` subcommand [here](https://warewulf.org/docs/v4.6.x/images/syncuser.html).

```shell
wwctl container syncuser --write rockylinux-nvidia-9
```

Once the `syncuser` command completes, we can assign this new image to a node. Assigning it directly to a node allows us to boot and test our newly created container before pushing it out to a profile. You can skip directly to assigning this to a profile if you desire.

```shell
wwctl node set node1 --container rockylinux-nvidia-9
```

Once you have tested your container image, you can assign the container to the necessary profile. In our example, we are sticking with the default profile:

```shell
wwctl profile set default --container rockylinux-nvidia-9
```

Finally, we can build the overlay:

```shell
wwctl overlay build
```

Upon reboot, your nodes will now log in with the newly created container image! You can find more information about Warewulf containers as well as more advanced configuration examples and instructions in the [Warewulf documentation](https://warewulf.org/docs/v4.6.x/images/images.html).

## References & related articles

[Warewulf Documentation](https://warewulf.org/docs/v4.6.x/images/images.html)  
[Warewulf Rocky Linux 9 NVIDIA Container Example](https://github.com/warewulf/warewulf-node-images/blob/main/examples/rockylinux-9-nvidia/Containerfile)  
[Hands on Warewulf: Solving Cluster Provisioning & Management](https://www.youtube.com/watch?v=7w8dAU9RSpE)  
[Warewulf: Deep Dive, Use Cases, and Examples](https://www.youtube.com/watch?v=EpVDeesAq4c)  


Install NVIDIA Drivers in a Warewulf Container


## Introduction
As mentioned [here](https://warewulf.org/docs/main/contents/ipmi.html):
It is possible to control the power or connect a console to your nodes being managed by Warewulf by connecting to the BMC through the use of IPMI

## Problem
The IPMI command works for all nodes after reboot, using their ipmitool command and node's IP address:
```
# ipmitool -I lanplus -H 10.0.0.11 -U root -P <PASSWORD> power status
Chassis Power is on
```

However, it does not work with wwctl power command:

```
wwctl power status cpu-20-10
ERROR  : cpu-20-10: Get Session Challenge command failed
Error: Unable to establish LAN session
Error: Unable to establish IPMI v1.5 / RMCP session
ERROR: exit status 1
```

## Resolution
You may need to configure IPMI to use **lanplus**. That could be achievable by setting `--ipmiinterface=lanplus`. Example:

```
wwctl profile set default \
> --netmask=255.255.255.0 --gateway=10.10.10.30 \
> --ipmiuser=root --ipmipass=[redacted] \
> --ipminetmask=255.255.255.0 --ipmiinterface=lanplus
```

## References & related articles
[Deploying a Dell PowerEdge and Cornelis Omni-Path Cluster with Warewulf](https://ciq.com/blog/deploying-a-dell-poweredge-and-cornelis-omni-path-cluster-with-warewulf/)

IPMI not working with wwctl power command for the LANPlus interface


## Introduction
If you are on an LTS version of Rocky Linux and would like to transition to the latest upstream version of Rocky Linux you need to revert the repositories and the release back to the default repositories

## Resolution
We can accomplish migrating from CIQ LTS for Rocky Linux 9.2 to Rocky Linux 9.4 running these commands:
 
```
sudo mtn dnf disable lts-9.2
sudo dnf swap ciq-lts92-rocky-release rocky-release --allowerasing --skip-broken
sudo dnf distrosync 
```

## Notes

- The dnf swap command will replace all of your Rocky repo's in /etc/yum/repos.d/ with the default repos. If you use any custom repo URLs, back them up before running this command.
- The dnf distrosync command will upgrade your server to 9.4. Please ensure you have the necessary backups before proceeding. 
- If you use secure boot, test this on a non production server first to ensure there are no issues.
- If dnf distrosync finishes but fails to install a package or two, you can run `dnf upgrade` to finish upgrading any missed packages.

## References & related articles
[CIQ Long Term Support](https://ciq.com/services/long-term-support/)

Migrate From Rocky Linux LTS to Rocky Linux


## Introduction

The Fuzzball Web GUI allows you to view logs from your running or completed job.

## Problem

Under some circumstances, a user may click on the "Logs" button within the job section of the
workflow dashboard and receive the message "No log output" even though the job completed
successfully and logs were expected. The problem may be that user does not have the appropriate
permissions to view the logs, or that the browser cannot access the API endpoint because Fuzzball
was installed using self-signed certificates.

## Symptoms

Here is an example showing the problem. The user has clicked on the "Logs" button for a workflow
that was expected to produce logs and is seeing the message "No log output".

![No log output in GUI](/images/articles/no-logs.png)

## Resolution

### Ensure you have permission to view logs

First, ensure that you have the appropriate permissions to view logs in the workflow.  If you are
viewing a job in a workflow submitted by another member of your shared account, you might need to
ask them for permission to view the logs.  They can add you as a job owner using the following CLI
command:

```text
$ fuzzball workflow add-owner <workflow UUID> <user UUID>
```

The workflow UUID and user UUID can be obtained with the `fuzzball workflow list` and `fuzzball
account list-members` commands respectively.  After you have been added as an owner you should be
able to view logs from jobs within the workflow.

### Self-signed certificate require exception

If you have the correct permission to view job logs, you should check to make sure that your browser
can access them from the API.  When self-signed certificates are used, the Fuzzball GUI asynchronous
access to the Fuzzball Orchestrate API can be blocked by your web browser's security policy, causing
an erroneous indication that no logs are available.

To correct this issue, you can direct your browser to the Fuzzball API endpoint.  The exact URI will
depend on the way that Fuzzball was installed, but it may be accessible by changing the string `ui`
in the Fuzzball URL to `api`.  For instance the following refers to the URL for the Fuzzball web GUI
and the API on an example installation respectively.

- https://ui.stable.fuzzball.ciq.dev/
- https://api.stable.fuzzball.ciq.dev/

In the case of a self-signed certificate, your browser may complain that the URL is unsafe and may
prompt you to create an exception.  After doing so, a browser refresh should allow you to view the
logs in the web GUI.


SSH Keys Issue on Rocky 9


## Introduction

A `mutex` (meaning `mutual exclusion`) prevents multiple threads from accessing the same resource simultaneously. When CIQ was running a `futex` test (a form of testing that checks the Linux kernel `futex` system call API), we observed `mutex` slowdown when running the `futex` testing on Rocky Linux 8.9 compared with CentOS 7.9. 

We investigated this performance discrepancy further and this article will reveal why there was a performance differential between the two operating systems.

## Resolution

The reason for the performance differences from both operating systems, was due to the BIOS settings on the HPE Blade machines that were used for testing. The CPU governor settings were set to `balanced power saving` mode. 

Upon configuring the CPU governor to `static high performance` mode in the BIOS, we saw a dramatic increase in performance for Rocky Linux 8.9.

Running the `futex` tests again on both Rocky Linux 8.9 and CentOS 7.9, produced the performance results as expected.

We also advise setting the `tuned-adm` profile to `hpc-compute` to ensure maximum performance from Rocky Linux.

## Conclusion

When comparing performance between a Rocky Linux X.X version and another operating system, CIQ advises to check the BIOS settings are set to have the CPU governor configured for maximum performance.

In addition, we advise setting the `tuned-adm` profile to `hpc-compute` to ensure maximum performance from Rocky Linux.

## References

* Futex Testing from the Linux Kernel Docs Team: https://www.kernel.org/doc/readme/tools-testing-selftests-futex-README

* General Mutex Subsystem from the Linux Kernel Docs Team: https://docs.kernel.org/locking/mutex-design.html

* tuned-adm man page: https://linux.die.net/man/1/tuned-adm

Performance Differences Between CentOS 7.9 and Rocky Linux 8.9 - a Root Cause Analysis and Recommendations


## Introduction

As mentioned [here](https://warewulf.org/docs/v4.5.x/contents/containers.html#):

*Since the inception of Warewulf over 20 years ago, Warewulf has used the model of the “Virtual Node File System” (VNFS) as a template image for the compute nodes. This is similar to a golden master image, except that the node file system exists within a directory on the Warewulf control node (e.g. a chroot()).*

*In hindsight, we’ve been using containers all along, but the buzzword just didn’t exist. Over the last 5-6 years, the enterprise has created a lot of tooling and standards around defining, building, distributing, securing, and managing containers, so Warewulf v4 now integrates directly within the container ecosystem to facilitate the process of VNFS image management.*

*If you are not currently leveraging the container ecosystem in any other way, you can still build your own chroot directories and use Warewulf as before.*

*It is important to understand that Warewulf is not running a container runtime on cluster nodes. While it is absolutely possible to run containers on cluster nodes, Warewulf is provisioning the container image to the bare metal and booting it. This container will be used as the base operating system and, by default, it will run entirely in memory. This means that when you reboot the node, the node retains no information about Warewulf or how it booted.*

## Problem

There could be cases where a node might not like to extract images for some reasons (BIOS settings, hardware specific problems, etc) and it is hard to find any solution to force iPXE for those nodes to pick the uncompressed images, and keep the other nodes using the compressed ones.

## Symptoms

When trying to provision the node with a compressed image, the server hangs during the PXE boot after downloading the container. After some minutes, the server might reboot with no warning message.

![Compressed Image](/images/articles/Compressedimage1.png)

## Resolution

There are two choices you can consider:

1. Probably the simpler method would be to copy `default.ipxe` to `nocompress.ipxe`, update it appropriately, and then direct nodes to use it with the ipxe template node attribute.
2. Define a node tag (something like `ipxe_nocompress`) and look for it in `default.ipxe`. You could then control whether to use compressed images or not based on what’s defined.


Pxe Boot with Compressed Container Images


## Introduction

When running a server that requires high availability, a user may not have the luxury to perform a reboot after making a configuration change.

The following guide will explain how to rename a network interface, without the need for a reboot.

## Problem

To change the names of network interfaces without rebooting the server.

## Resolution

### Prerequisites

Access to the `root` user account or a user with escalated privileges.

### Method 1 - ip link set

To temporarily change the name of an interface, use the `ip link set` command.

Bring the network interface that you wish to rename down:

```bash
sudo ip link set <INTERFACE_NAME> down
```

Rename the interface:

```bash
sudo ip link set <INTERFACE_NAME> name <NEW_INTERFACE_NAME>
```

Enable the newly renamed interface:

```bash
sudo ip link set <NEW_INTERFACE_NAME> up
```

**Please note your changes will be lost if you reboot your server.**

### Method 2 - udev rules by MAC Address

Check under `/etc/udev/rules.d/` that you have a `70-persistent-net.rules` file. If the file is not there, create it with `touch /etc/udev/rules.d/70-persistent-net.rules`

Confirm the interface you wish to edit by using `ip -brief address show`. You will see a similar output as in the below example:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# ip -brief address show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
enp8s0           UP             <PUBLIC_IP>/24
```

Open `/etc/udev/rules.d/70-persistent-net.rules` with your text editor of choice and modify the `NAME` field of the interface you wish to change. A before and after example follows:

Before:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="<PUBLIC_IP_HERE>", NAME="enp8s0"
```

After:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="<PUBLIC_IP_HERE>", NAME="new-name-1"
```

Bring the network interfaces down with `ip link set <INTERFACE_NAME> down`. In the author's case, the interface was `enp8s0`:

```bash
ip link set enp8s0 down 
```

Use the `udevadm control` command to reload the udev rules and request device events from the kernel with `udevadm trigger`:

```bash
udevadm control --reload-rules; udevadm trigger --subsystem-match=net --action=add
```

Confirm that the network interface was renamed with `ip -brief address show`. The author correctly found this to be the case:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# ip -brief address show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
new-name-1       DOWN           <PUBLIC_IP>/24
```

Bring the link back up using `ip link set <NEW_INTERFACE_NAME> up`. Please see the following example:

```bash
ip link set new-name-1 up 
```

Observe that the interface is up and has been renamed without requiring a reboot:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# ip -brief address show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
new-name-1       UP             <PUBLIC_IP>/24
```

Even if the machine is rebooted, the changes will still be present.

## Conclusions

With the methods above, you should be able to rename a network interface and avoid having to reboot.

## References & related articles

[udevadm man page](https://linux.die.net/man/8/udevadm)

[udev rules introduction](https://opensource.com/article/18/11/udev)


Renaming Network Interfaces Without Rebooting


## Introduction
This article addresses the issue where the Sriov configuration script runs manually but fails when executed automatically by systemd. The root cause is identified as a timing or dependency issue.

## Problem
The script `/etc/sriov/setup_dpdk_nic.sh` works when executed manually after a node restart but fails when run automatically by systemd. This indicates a potential timing or dependency issue with the service startup.

## Symptoms
- Script executes successfully when run manually after node restart.
- Script fails when triggered automatically by systemd during the boot process.

## Resolution
Adjust the service unit file to address potential timing and dependency issues. Update the service unit file as follows:

```ini
[Unit]
Description=Sriov Config Service
After=network-online.target systemd-udevd.service
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/etc/sriov/setup_dpdk_nic.sh
RemainAfterExit=yes
TimeoutStopSec=90
#StartLimitInterval=120
#StartLimitBurst=3

[Install]
WantedBy=multi-user.target
```

### Explanation of Changes:
1. **[Unit] Section:**
   - **After=network-online.target systemd-udevd.service:** Ensures the service starts only after the network is fully online and systemd-udevd is running. This is crucial for NIC-related configurations.
   - **Wants=network-online.target:** Ensures the network is available when the service runs.

2. **[Service] Section:**
   - **RemainAfterExit=yes:** Keeps the service active after the script completes, aiding in tracking and status reporting.
   - **TimeoutStopSec=90:** Sets a timeout for the stop operation.
   - **StartLimit* Parameters:** Commented out as they control service restart limits, which are less relevant for a oneshot service.

### Benefits:
- Ensures the NIC configuration script runs at the correct time during the boot process.
- Improves service status reporting by keeping the service active after script execution.

### Troubleshooting:
If the issue persists, provide the output of the following command for further review:
```sh
journalctl -u sriov-configure.service
```

## Root Cause
The problem likely stemmed from the script running before the network was fully online. By adjusting the service dependencies, we ensure the script runs at the appropriate time during the boot sequence.

Resolving Timing and Dependency Issues in Systemd Service Unit for Sriov Configuration


## Introduction

Dual-booting is the practice of installing and using two separate operating systems on one computer. This setup enables you to select which operating system to boot into during startup, offering the flexibility to switch between them as needed.

## Problem

After installing the August 2024 Windows security update, [KB5041585](https://support.microsoft.com/help/5043076) or the August 2024 preview update, you might face issues with booting Linux if you have enabled a dual-boot setup of Windows and Linux on your machine. The result of this issue, is your device might fail to boot Linux and show the following error message:

```shell
Verifying shim SBAT data failed: Security Policy Violation
Something has gone seriously wrong: SBAT self-check failed: Security Policy Violation
```

## Resolution

Please follow the instructions highlighted in [this Microsoft article](https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-23h2#august-2024).

## Root Cause

The August 2024 Windows security and preview updates apply a Secure Boot Advanced Targeting (SBAT) setting to machines that run Windows. This blocks old and vulnerable boot managers. This SBAT update will not be applied to devices where dual-booting is detected. On some machines, the dual-boot detection did not detect some customized methods of dual-booting and applied the SBAT value when it shouldn't have done so.

---
**_IMPORTANT:_** _This known issue only occurs with the installation of the August 2024 security and preview updates. The September 2024 [KB5043076](https://support.microsoft.com/en-us/topic/september-10-2024-kb5043076-os-builds-22621-4169-and-22631-4169-215aad1e-3f3f-44bd-9868-91a2bd450a07) and later security updates do not contain the settings that cause this issue. If you install the September 2024 update, you don’t need to apply the workaround listed above._

---

## Notes

- Disabling Secure Boot is a temporary workaround, but is not recommended for long-term security.
- Ensure your system firmware (BIOS/UEFI) is up to date before applying updates.
- Monitor Windows Update for any future patches addressing this issue.

## References & Related Articles (Optional)

- [Microsoft August 2024 Update Support Article](https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-23h2#august-2024)


Rocky Linux Fails to Boot in a Dual-Boot Setup with Windows


## Introduction
In certain environments, managing user namespace mappings across multiple hosts is essential for consistent access control and permissions. A common approach is to symlink /etc/subuid and /etc/subgid files to a shared storage location to easily distribute these mappings. However, this approach can lead to issues with tools like Apptainer that utilize user namespaces, as they rely on newuidmap and newgidmap utilities that do not support symlinks. This article outlines the problem, details the root cause, and offers alternative solutions to achieve synchronization across hosts without symlinking.

## Problem
To simplify the management of `/etc/subuid` and `/etc/subgid`, you may attempt to symlink these files to a shared storage. This approach is intended to ensure consistent mapping across multiple hosts.

## Symptoms
When attempting to build with Apptainer in fakeroot mode while using symlinked `subuid` and `subgid` files, you may encounter the following error:

```
# apptainer build --fakeroot rockylinux_fakeroot.sif docker://rockylinux:9
ERROR  : 'newgidmap' execution failed. Check that 'newgidmap' is setuid root or has setcap cap_setgid+eip.
ERROR  : Error while waiting event for user namespace mappings: no event received
```

## Root Cause
This issue occurs because `newuidmap` and `newgidmap` utilities do not support symbolic links. When these programs attempt to read the `subuid` and `subgid` files, they fail due to the `O_NOFOLLOW` flag in the `openat` system call, which prevents following symbolic links. The following strace output illustrates this behavior:

```
openat(AT_FDCWD, "/etc/subgid", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW) = -1 ELOOP (Too many levels of symbolic links)
```
The `O_NOFOLLOW` flag, as specified in the `open` man page, causes the call to fail with an _ELOOP_ error if the final path component is a symbolic link, which is what happens here.

## Resolution
To synchronize `subuid` and `subgid` files across hosts without using symlinks, consider alternative methods such as:

1. **Rsync and Cron Jobs**: Use `rsync` in conjunction with a cron job to periodically sync these files across hosts. For details, refer to the [Rocky Linux Rsync Guide](https://docs.rockylinux.org/guides/backup/rsync_ssh/) and [Rocky Linux Cron Guide](https://docs.rockylinux.org/guides/automation/cron_jobs_howto/).

2. **Ansible Automation**: Automate file synchronization with Ansible for a more flexible and scalable solution. Ascender, an orchestration tool from CIQ, can assist in managing these automation tasks. For more details, refer to the [Rocky Linux Ansible Guide](https://docs.rockylinux.org/books/learning_ansible/02-advanced/) and [Ascender Automation](https://ciq.com/products/ascender/).

## References & related articles
Rocky Linux Rsync Guide [https://docs.rockylinux.org/guides/backup/rsync_ssh/]  
Rocky Linux Cron Guide [https://docs.rockylinux.org/guides/automation/cron_jobs_howto/]  
Rocky Linux Ansible Guide [https://docs.rockylinux.org/books/learning_ansible/02-advanced/]  
Ascender Automation [https://ciq.com/products/ascender/]  

Error When Symlinking Subuid And Subgid


## Introduction

At CIQ, customers will often create tickets due to a kernel crash they experienced. As part of the investigation process, the Support Team asks for a `vmcore` dump to be uploaded to CIQ's Secure File Exchange (SFE) service for analysis.

For more information on how to upload a `vmcore` dump or other files to the SFE service, please consult the [How to Upload Files to the CIQ Secure File Exchange (SFE) Service](https://kb.ciq.com/article/how-load-files-using-ciq-secure-file-exchange) KB article.

This article will detail some of the steps that can be taken with the Crash Utility for `vmcore` dump analysis.

For further reading, the author highly recommends the following resources:

* [Crash Utility White Paper by David Anderson](https://crash-utility.github.io/crash_whitepaper.html)

* [Linux Kernel Crash Book - Everything you need to know by Igor Ljubuncic](https://www.dedoimedo.com/computers/www.dedoimedo.com-crash-book.pdf)

## Example issue

A customer created a ticket with CIQ, after their CentOS 7.9 node using CIQ Bridge had a kernel panic and rebooted. What follows will be how to setup the Crash Utility and the steps taken to debug the issue.

### Crash Utility environment setup

* Set up a Rocky Linux 9.x installation on a node size of your choice.

* Update all packages:

```bash
sudo dnf update -y
```

* Install the `kexec-tools` package:

```bash
sudo dnf install -y kexec-tools 
```

* From the node that went into a kernel panic state, find the kernel version:

```bash
uname -r
```

* With the kernel version identified, search and download the equivalent version of the `kernel-debuginfo-common` package. In the CentOS 7.9 example listed above, the kernel version was `3.10.0-1160.119.1.el7.x86_64`, thus the required package was `kernel-debuginfo-common-x86_64-3.10.0-1160.119.1.el7.x86_64.rpm`.

* Similarly, the same version of the `kernel-debuginfo` package is also needed. Again referencing the CentOS 7.9 example, downloading the `kernel-debuginfo-3.10.0-1160.119.1.el7.x86_64.rpm` package is required.

* **Note for Rocky Linux users** - to find the appropriate `kernel-debuginfo-common` and `kernel-debuginfo` packages, go to [The Rocky Linux Vault](https://dl.rockylinux.org/vault/rocky/). Using Rocky Linux 9.5 with the x86_64 architecture as an example, the `kernel-debuginfo-common` package is found [in the devel repository](https://dl.rockylinux.org/vault/rocky/9.5/devel/x86_64/os/Packages/k/) and `kernel-debuginfo` [is in the devel repository under debug](https://dl.rockylinux.org/vault/rocky/9.5/devel/x86_64/debug/tree/Packages/k/).

* Install both `kernel-debuginfo-common` and `kernel-debuginfo` packages using the `localinstall` command. The order the packages are installed in also matters:

```bash
dnf localinstall -y <path_to_kernel-debuginfo-common_rpm>
dnf localinstall -y <path_to_kernel-debuginfo_rpm>
```

* **Note for CentOS users** - the `vmcore` dump will be generated as a `vmcore.flat` file. This is not natively readable by the Crash Utility and thus has to be converted into a `vmcore` dump file. Perform the following command to convert the file:

```bash
/usr/sbin/makedumpfile -R ./vmcore < ./vmcore.flat
```

* Finally, start the Crash Utility by setting the `vmlinux` version as an argument under your `/usr/lib/debug/lib/modules/<kernel_version>` directory and pointing to the `vmcore` dump file:

```bash
crash /usr/lib/debug/lib/modules/<kernel_version>/vmlinux <path_to_vmcore_dump>
```

### Example crash dump analysis

* Starting the Crash Utility will bring up an overview of the `vmcore` dump, including the date the dump was taken, node name, memory availability, kernel version, and the kernel panic reason.

* An example kernel panic that CIQ often sees from customers running Kubernetes is `blocked tasks`:

```bash
"Kernel panic - not syncing: hung_task: blocked tasks"
```

* From the initial overview, the amount of memory available was plentiful during the kernel panic. However, as you will see later on in the article, having plentiful memory does not rule out a memory-related issue:

```bash
MEMORY: 1011.7 GB
```

* For deeper analysis on memory usage and availability, the `kmem -i` command can be ran and produces an output similar to the below example:

```bash
crash> kmem -i
                 PAGES        TOTAL      PERCENTAGE
    TOTAL MEM  261010406     995.7 GB         ----
         FREE  228447706     871.5 GB   87% of TOTAL MEM
         USED  32562700     124.2 GB   12% of TOTAL MEM
       SHARED  1626752       6.2 GB    0% of TOTAL MEM
      BUFFERS     3277      12.8 MB    0% of TOTAL MEM
       CACHED  20211378      77.1 GB    7% of TOTAL MEM
         SLAB   765701       2.9 GB    0% of TOTAL MEM

   TOTAL HUGE        0            0         ----
    HUGE FREE        0            0    0% of TOTAL HUGE

   TOTAL SWAP  1048575         4 GB         ----
    SWAP USED        0            0    0% of TOTAL SWAP
    SWAP FREE  1048575         4 GB  100% of TOTAL SWAP

 COMMIT LIMIT  131553778     501.8 GB         ----
    COMMITTED  26620006     101.5 GB   20% of TOTAL LIMIT
```

* Running the `sys -i` command shows more details about the hardware, as well as BIOS information, and product tags. Sensitive information such as serial numbers have been removed in the following example:

```bash
crash> sys -i
        DMI_BIOS_VENDOR: HPE
       DMI_BIOS_VERSION: A43
          DMI_BIOS_DATE: 12/03/2021
         DMI_SYS_VENDOR: HPE
       DMI_PRODUCT_NAME: ProLiant DL325 Gen10 Plus
    DMI_PRODUCT_VERSION:
     DMI_PRODUCT_SERIAL: -
       DMI_PRODUCT_UUID: -
       DMI_BOARD_VENDOR: HPE
         DMI_BOARD_NAME: ProLiant DL325 Gen10 Plus
      DMI_BOARD_VERSION:
       DMI_BOARD_SERIAL: -
    DMI_BOARD_ASSET_TAG: -
     DMI_CHASSIS_VENDOR: HPE
       DMI_CHASSIS_TYPE: 23
    DMI_CHASSIS_VERSION:
     DMI_CHASSIS_SERIAL: -
  DMI_CHASSIS_ASSET_TAG: -
```

* The `bt` command produces a backtrace output for the process that caused the kernel panic. The succeeding example shows process `650` running command `khungtaskd` was responsible for the kernel panic:

```bash
crash> bt
PID: 650      TASK: ffff9b2ffbb59080  CPU: 47   COMMAND: "khungtaskd"
 #0 [ffff9b2ffb663cb0] machine_kexec at ffffffffb2c69854
 #1 [ffff9b2ffb663d10] __crash_kexec at ffffffffb2d29f12
 #2 [ffff9b2ffb663de0] panic at ffffffffb33ab713
 #3 [ffff9b2ffb663e60] watchdog at ffffffffb2d56bfe
 #4 [ffff9b2ffb663ec8] kthread at ffffffffb2ccb621
```

* More information on `khungtaskd` can be found [at this article](https://www.tencentcloud.com/document/product/213/41974), however a summary can be observed beneath:

> The kernel hung task is based on a single kernel thread named as khungtaskd, which monitors processes in the TASK_UNINTERRUPTIBLE status. If a process stuck in D state during the period specified by kernel.hung_task_timeout_secs (defaults to 120 seconds), the stack information of this hung task process will be printed.

* To verify that the kernel is configured to crash in a `khungtaskd` state, the `p` command (to print the value of an expression) with the `sysctl_hung_task_panic` argument can be used:

```bash
crash> p sysctl_hung_task_panic
sysctl_hung_task_panic = $1 = 1
```

* Similarly the timeout can also be checked using the `p sysctl_hung_task_timeout_secs` command:

```bash
crash> p sysctl_hung_task_timeout_secs
sysctl_hung_task_timeout_secs = $2 = 360
```

* To view more information about a particular process, the `ps` command plus the PID number as an argument can be used:

```bash
crash> ps 650
      PID    PPID  CPU       TASK        ST  %MEM      VSZ      RSS  COMM
>     650       2  47  ffff9b2ffbb59080  RU   0.0        0        0  [khungtaskd]
```

* Finding out more information about which tasks were blocked, can be done with the `log` command and a `grep` for the word `INFO`. In the subsequent example, `systemd` is seen as being blocked:

```bash
crash> log | grep INFO
[3970064.996286] INFO: task systemd:1 blocked for more than 360 seconds.
```

* For showing the stack traces of all blocked tasks in a `TASK_UNINTERRUPTIBLE` state, the `foreach UN bt` command is recommended:

```bash
foreach UN bt
```

* Further filtering is also possible on all blocked tasks with this command string. The command picks tasks #2, #3, and #4 from each process, places them into three columns, counts the amount of stack trace messages occurrences and sorts from highest to lowest:

```bash
foreach UN bt | awk '/#[2-4] /{print $3,$5}' | paste - - - | sort | uniq -c | sort -nr
```

* An example is here:

```bash
crash> foreach UN bt | awk '/#[2-4] /{print $3,$5}' | paste - - - | sort | uniq -c | sort -nr
     46 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     proc_cgroup_show ffffffffb2d35146
     33 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     cgroup_lock_live_group ffffffffb2d2ed49
      2 __mutex_lock_killable_slowpath ffffffffb33b6520 mutex_lock_killable ffffffffb33b65fe    iterate_dir ffffffffb2e71d00
      1 schedule_timeout ffffffffb33b5831       wait_for_completion ffffffffb33b805d    wait_rcu_gp ffffffffb2cc828e
      1 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     proc_cgroupstats_show ffffffffb2d2f2be
      1 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     cgroup_release_agent ffffffffb2d2ffe5
      1 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     cgroup_free_fn ffffffffb2d34308
```

* From the above output, a high volume of `mutex_lock_slowpath` messages can be viewed. Researching the error attributes this to either a hardware issue or low resource availability (CPU / Memory / I/O).

* The full list of unique blocked tasks can be found by running `foreach UN bt | grep COMMAND | awk '{print $NF}' | sort -u`:

```bash
crash> foreach UN bt | grep COMMAND | awk '{print $NF}' | sort -u
"cadvisor"
"dockerd"
"filebeat"
"kubelet"
"kworker/113:0"
"kworker/31:1"
"process-exporte"
"runc"
"systemd"
"systemd-journal"
```

* To dig further into a blocked task, use the `set` command with the PID as the argument. The ensuing example sets the focus on the blocked task `systemd`:

```bash
crash> set 1
    PID: 1
COMMAND: "systemd"
   TASK: ffff9a32db050000  [THREAD_INFO: ffff9a32dbb9c000]
    CPU: 42
  STATE: TASK_UNINTERRUPTIBLE
```

* Then run `bt -l` for more verbose backtrace messages. In the above-mentioned `mutex_lock_slowpath` example, the process that was running before the `mutex_lock` was engaged was `proc_cgroup_show`:

```bash
crash> bt -l
#2 [ffff9a32dbb9fda8] __mutex_lock_slowpath at ffffffffb33b6ca7
    /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/include/linux/spinlock.h: 337
#3 [ffff9a32dbb9fe08] mutex_lock at ffffffffb33b602f
    /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/arch/x86/include/asm/current.h: 14
#4 [ffff9a32dbb9fe20] proc_cgroup_show at ffffffffb2d35146
    /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/kernel/cgroup.c: 4706
```

* The line number from the `cgroup.c` file being referenced by `proc_cgroup_show` was `4706`. It is possible to view the contents of the source code in the `vmcore` dump with the `list` or `l` command:

```bash
crash> l /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/kernel/cgroup.c: 4706
4701
4702            retval = 0;
4703
4704            mutex_lock(&cgroup_mutex);
4705
4706            for_each_active_root(root) {
4707                    struct cgroup_subsys *ss;
4708                    struct cgroup *cgrp;
4709                    int count = 0;
4710
```

* You can see from line `4704` that the `mutex_lock` is related to the `Cgroup` system. From the [kernel.org documentation on Cgroups](https://docs.kernel.org/admin-guide/cgroup-v1/cgroups.html):

> There is a global mutex, cgroup_mutex, used by the cgroup system. This should be taken by anything that wants to modify a cgroup. It may also be taken to prevent cgroups from being modified, but more specific locks may be more appropriate in that situation.

* Since `Cgroups` handle the allocation of resources, the mutex lock observed here points towards a lack of resources on the node.

* As an aside, if the `mutex_lock` code pointed to `mutex_lock(&dentry->d_inode->i_mutex);`, that would highlight an inode-related error on the storage medium and further investigation would be required there.

* Finally, after running the `log` command to dump the `system message buffer`, multiple `Hardware Error` messages were observed related to the customer's memory DIMMs:

```bash
crash> log
<output_redacted>
[3968484.124456] {176586}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 20480
[3968484.124460] {176586}[Hardware Error]: It has been corrected by h/w and requires no further action
[3968484.124461] {176586}[Hardware Error]: event severity: corrected
[3968484.124463] {176586}[Hardware Error]:  Error 0, type: corrected
[3968484.124464] {176586}[Hardware Error]:  fru_text: DIMM# Sourced
[3968484.124465] {176586}[Hardware Error]:   section_type: memory error
[3968484.124466] {176586}[Hardware Error]:   error_status: 0x0000000000040400
[3968484.124468] {176586}[Hardware Error]:   physical_address: 0x000000f877d52000
[3968484.124470] {176586}[Hardware Error]:   node: 0 card: 3 module: 1 rank: 2 bank: 6 row: 57311 column: 0
[3968484.124471] {176586}[Hardware Error]:  Error 1, type: corrected
[3968484.124472] {176586}[Hardware Error]:  fru_text: DIMM# Sourced
[3968484.124473] {176586}[Hardware Error]:   section_type: memory error
[3968484.124474] {176586}[Hardware Error]:   error_status: 0x0000000000040400
[3968484.124475] {176586}[Hardware Error]:   physical_address: 0x000000f7f9b63180
[3968484.124477] {176586}[Hardware Error]:   node: 0 card: 3 module: 1 rank: 2 bank: 6 row: 56806 column: 16
[3968484.124501] [Hardware Error]: Corrected error, no action required.
[3968484.125383] [Hardware Error]: CPU:0 (17:31:0) MC1_STATUS[-|CE|-|-|AddrV|-|-|-|-]: 0x940000000000009f
[3968484.126221] [Hardware Error]: Error Addr: 0x000000f877d52000, IPID: 0x0000000000000000
[3968484.126907] [Hardware Error]: Instruction Fetch Unit Extended Error Code: 0
[3968484.127593] [Hardware Error]: Instruction Fetch Unit Error: microtag probe port parity error.
[3968484.128280] [Hardware Error]: cache level: L3/GEN, tx: RESV
```

## Example issue resolution

Ultimately it was found that the lack of resources were due to `Hardware Errors` in the customer's memory DIMMs, thus causing a contention for resources resulting in tasks being blocked for over 360 seconds and therefore invoking a kernel panic.

## Conclusion

`vmcore` dump analysis is a vast and complicated process. There are almost a limitless number of ways that the kernel can crash and the root cause may not always be the same from system to system. The author hopes that the example analysis above for blocked tasks can help to demystify some of the research required and make it easier to identify a root cause if you run into a similar situation.

Tips and Tricks for Analyzing a vmcore dump with the Crash Utility


## Introduction

If no certificates are supplied by the Administrator during the configuration and installation
procedure, the Fuzzball Operator will create self-signed certificates for hosting Fuzzball
Orchestrate.  

## Problem

You may run into issues using the Fuzzball CLI if the cluster (context) you are accessing is using
a self-signed certificate.

## Symptoms

If you receive an error like the following when attempting to use the Fuzzball CLI it suggests that
your environment is not properly configured to recognize the self-signed TLS certificates in use.

```text
tls: failed to verify certificate: x509: certificate signed by unknown authority
```

## Resolution

An administrator can use the following commands to export the relevant certificates from the
Fuzzball K8s installation:

```text
# mkdir certs

# kubectl get secret -n cert-manager root-ca-cert -o "jsonpath={.data['ca\.crt']}" | base64 --decode >certs/ca.crt

# kubectl get secret -n cert-manager root-ca-cert -o "jsonpath={.data['tls\.crt']}" | base64 --decode >certs/tls.crt
```

These certificates can be distributed to the systems where the Fuzzball CLI is installed and then
used by adding an environment variable like so:

```text
$ export SSL_CERT_DIR=/path/to/certs
```

Alternately, CLI users can set an environment variable, `FUZZBALL_INSECURE=true` or use the
`--insecure` flag when invoking the Fuzzball command.

## Notes

The solutions selected here are appropriate for development clusters, testing, and debugging, but it
is ultimately more secure for administrators to use certificates issued by 3rd parties.  The
Fuzzball documentation contains a section describing how to do this.


Using the CLI with Self-Signed TLS Certificates


## Introduction
As explained in this [guide](https://warewulf.org/docs/main/contents/provisioning.html), the only thing that Warewulf requires to provision is that the node is set to PXE boot. You may need to change the boot order if there is a local disk present and bootable. This is a configuration change you will have to make in the BIOS of the cluster node, as this is the first step of the provisioning process. 
Once the nodes boot to PXE, PXE will request a BOOTP/DHCP address on the network, the Warewulf controller’s DHCP server will respond with a network configuration and filename to try and boot, PXE will attempt to download the filename referred to in the DHCP response via TFTP. 
The downloaded file will execute an iPXE stack which will reach out to the Warewulf server for it’s configuration. Then, the Warewulf server will generate the iPXE configuration which will include directions of what else is necessary to download and how to boot. 
The kernel, container image, kernel modules, and system overlay are all downloaded over REST HTTP from the Warewulf Server. iPXE executes the kernel and processes the overlays to provide a unified root file system. 
Warewulf bootstraps the initialization of cluster node’s operating system: File System (re)configuration, SELinux, and `wwclient` is called as a background daemon and sleeps until network is ready. And for last, the Warewulf bootstrap execs the container’s `/sbin/init`


## Problem
Node(s) are not booting after set to boot from PXE.

## Resolution
The resolution for this problem is not straight forward as this problem could be caused by several different parts of the booting process. Therefore is good starting with a sanity check
 
* Ensure httpd, dhcpd, tftp-server.socket (or xinetd) are all enabled and running

`systemctl status <service>`

* Ensure firewall rules are set or disabled. Is good to stop firewall completely to rule out any firewall problems

`systemctl stop firewalld`

* Check Selinux. If Selinux is enforcing check [this guide](https://warewulf.org/docs/main/contents/security.html) to enable SELinux if needed, or just disable SELinux to rule out any SELinux issues.

`setenforce 0`

### Test TFTP Server.
Another option to check is making sure TFTP service is working properly. 
We need to make sure the server is able to get a file from warewulf directory
`tftp <tftpserverIPADDR> -c get /var/lib/warewulf/bin-x86_64-efi-snponly.efi`

**Note:** This usually only test the tftp server is working properly but to discard any network issue, we advice to run the command from a node in the same network as the warewulf headnode as well, and check if it gets the file successfuly from the tftp server.


## References & related articles
[Node Provisioning](https://warewulf.org/docs/main/contents/provisioning.html)


Troubleshooting PXE Booting, Node Provisioning.


## Introduction
This article addresses the issue of delayed errata page publication in the Rocky Linux repository.

## Problem
The errata page for a specific package update may not be immediately available after the package is made available in the repository. This delay can occur due to various administrative or procedural reasons within the maintenance and release management process.

## Resolution
If you encounter a delayed errata page publication, follow these steps:

1. **Check the package update history**: Verify that the package update was recently made available in the repository.
2. **Verify the errata page status**: Check the Rocky Linux forums or other community resources for information on the status of the errata page.
3. **Wait for the errata page to be published**: The errata page may be in the process of being updated or created. Wait for the page to be published before attempting to access it.
4. **Contact CIQ Support**: If the issue persists, contact CIQ Support for further assistance.

Note: The errata page publication process is owned by the open source community, and delays may occur due to various reasons. Be patient and allow sufficient time for the errata page to be published.

Troubleshooting Missing Errata Pages in Rocky Linux Repository


## Introduction

[`udev`](https://linux.die.net/man/8/udev) provides a dynamic device directory containing only the files for actually present devices. It creates or removes device node files usually located in the `/dev` directory, or it renames network interfaces.

As part of the hotplug subsystem, `udev` is executed if a kernel device is added or removed from the system. On device creation, `udev` reads the `sysfs` directory of the given device to collect device attributes such as label, serial number or bus device number. These attributes may be used as keys to determine a unique name for the device.

On another hand, RAID devices are virtual devices created from two or more real block devices. This allows multiple devices (typically disk drives or partitions thereof) to be combined into a single device to hold, for example, a single filesystem. Some RAID levels include redundancy and so can survive some degree of device failure. Linux Software RAID devices are implemented through the `md` (Multiple Devices) device driver. More information can be found [at the mdadm man page.](https://linux.die.net/man/8/mdadm)

## Problem

The user may have created `udev` rules to handle the RAID device creation, causing the `md0_sync` process to hang.

## Symptoms

*Please note that the replication of this issue is random, as the race condition does not happen every time. You may have to run the below `mdam` command a few times, in order to recreate the race condition.*

The following udev rules were used:

```bash
SUBSYSTEM=="block",ACTION=="add|change",KERNEL=="md*",\
ATTR{md/sync_speed_max}="2000000",\
ATTR{md/group_thread_cnt}="64",\
ATTR{md/stripe_cache_size}="8192",\
ATTR{queue/nomerges}="2",\
ATTR{queue/nr_requests}="1023",\
ATTR{queue/rotational}="0",\
ATTR{queue/rq_affinity}="2",\
ATTR{queue/scheduler}="none",\
ATTR{queue/add_random}="0",\
ATTR{queue/max_sectors_kb}="4096"
```

Then to create the RAID device, the following command was run:

```bash
$ mdadm --create --verbose --chunk=128 --level=6 -n 4 /dev/md0 /dev/vda /dev/vdb /dev/vdc /dev/vdd
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: size set to 52395008K
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
```

You can check the status of the RAID creation by checking the `/proc/mdstat` file:

```bash
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 vdd[3] vdc[2] vdb[1] vda[0]
      104790016 blocks super 1.2 level 6, 128k chunk, algorithm 2 [4/4] [UUUU]
      [>....................]  resync =  4.2% (2203904/52395008) finish=3.7min speed=220390K/sec
```

When you check for a second time, you may find the file is empty or the `resync` completion percentage status has not moved.

You could find the following errors in the logs:

```bash
Sep 11 18:39:21 rockylinux.com systemd-udevd[3394]: nvme2n1: /etc/udev/rules.d/udev.rules:53 Failed to write ATTR{/sys/devices/pci0000:60/0000:60:03.3/0000:65:00.0/nvme/nvme2/nvme2n1/queue/max_sectors_kb}, ignoring: Invalid argument
Sep 12 10:22:33 rockylinux.com systemd-udevd[570894]: nvme2n1: /etc/udev/rules.d/udev.rules:53 Failed to write ATTR{/sys/devices/pci0000:60/0000:60:03.3/0000:65:00.0/nvme/nvme2/nvme2n1/queue/max_sectors_kb}, ignoring: Invalid argument
Sep 12 10:43:13 rockylinux.com systemd-udevd[3450]: nvme2n1: /etc/udev/rules.d/udev.rules:53 Failed to write ATTR{/sys/devices/pci0000:60/0000:60:03.3/0000:65:00.0/nvme/nvme2/nvme2n1/queue/max_sectors_kb}, ignoring: Invalid argument
Sep 12 12:21:44 rockylinux.com systemd-udevd[65370]: nvme2n1: /etc/udev/rules.d/udev.rules:53 Failed to write ATTR{/sys/devices/pci0000:60/0000:60:03.3/0000:65:00.0/nvme/nvme2/nvme2n1/queue/max_sectors_kb}, ignoring: Invalid argument
```

The `md0_resync` task will hang with the following logs (*it may cause the kernel to panic and crash, if you have `kernel.hung_task_panic = 1` included in your `/etc/sysctl.conf` file*):

```bash
Sep 10 10:40:38 rockylinux.com kernel: INFO: task md0_resync:2365161 blocked for more than 122 seconds.
Sep 10 10:40:38 rockylinux.com kernel:      Tainted: G S      W6.6.32-120240613.el9.x86_64 #1
Sep 10 10:40:38 rockylinux.com kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 10 10:40:38 rockylinux.com kernel: task:md0_resync      state:D stack:0     pid:2365161 ppid:2      flags:0x00004000
Sep 10 10:40:38 rockylinux.com kernel: Call Trace:
Sep 10 10:40:38 rockylinux.com kernel: <TASK>
Sep 10 10:40:38 rockylinux.com kernel: __schedule+0x222/0x660
Sep 10 10:40:38 rockylinux.com kernel: schedule+0x5e/0xd0
Sep 10 10:40:38 rockylinux.com kernel: md_do_sync+0xcef/0x1030
Sep 10 10:40:38 rockylinux.com kernel: ? sched_clock_cpu+0xf/0x190
Sep 10 10:40:38 rockylinux.com kernel: ? membarrier_register_private_expedited+0xa0/0xa0
Sep 10 10:40:38 rockylinux.com kernel: md_thread+0xb0/0x160
Sep 10 10:40:38 rockylinux.com kernel: ? super_90_load.part.0+0x350/0x350
Sep 10 10:40:38 rockylinux.com kernel: kthread+0xe3/0x110
Sep 10 10:40:38 rockylinux.com kernel: ? kthread_complete_and_exit+0x20/0x20
Sep 10 10:40:38 rockylinux.com kernel: ret_from_fork+0x31/0x50
Sep 10 10:40:38 rockylinux.com kernel: ? kthread_complete_and_exit+0x20/0x20
Sep 10 10:40:38 rockylinux.com kernel: ret_from_fork_asm+0x11/0x20
Sep 10 10:40:38 rockylinux.com kernel: </TASK>
```

## Resolution

To conclude from the above findings, the `udev` rule for the RAID device may be triggering changes too early, leading to a race condition during the `md0_resync` task. Removing this rule fixes the issue.

One potential solution is to use the [`udevadm settle`](https://linux.die.net/man/8/udevadm) approach or introduce a delay to avoid the race condition from occurring.

Alternatively, monitoring the `md0` status before applying devices changes via a script, could help to ensure the `resync` completes without `udev` interference.

## Root Cause

This issue is related to common `udev` limitations. At the time of a `udev` query, some storage devices might not be accessible. There might be a delay between event generation and processing, especially with numerous devices involved, causing a lag between kernel detection and link availability. External programs invoked by `udev` rules, such as `blkid`, might briefly open the device, making it temporarily inaccessible for other tasks.

## References & related articles

[`udev` man page](https://linux.die.net/man/8/udev)  
[`mdadm` man page](https://linux.die.net/man/8/mdadm)  
[`udevadm` man page](https://linux.die.net/man/8/udevadm)


Udev Rules Causing RAID Device Creation to Hang


## Introduction

[Security Technical Implementation Guides (STIGs)](https://www.titania.com/resources/guides/disa-stig-compliance-explained) produced by the Defense Information Systems Agency (DISA) are heavily implemented in the Enterprise Linux ecosystem. The configurations provided allow for hardening and mitigation against cybersecurity threats.

When DISA STIG is implemented in Rocky Linux, there are occasions where you see messages in `dmesg` about reduced security. This article will explain one such message in `unhashed kernel memory addresses` and if any action is needed to be taken.

## Problem

When looking at `dmesg` you see a warning: *This system shows unhashed kernel memory addresses*.

## Symptoms

The following messages are shown:

```text
[    0.012710] **********************************************************
[    0.012710] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[    0.012711] **                                                      **
[    0.012711] ** This system shows unhashed kernel memory addresses   **
[    0.012711] ** via the console, logs, and other interfaces. This    **
[    0.012712] ** might reduce the security of your system.            **
[    0.012712] **                                                      **
[    0.012712] ** If you see this message and you are not debugging    **
[    0.012712] ** the kernel, report this immediately to your system   **
[    0.012713] ** administrator!                                       **
[    0.012713] **                                                      **
[    0.012713] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[    0.012713] **********************************************************
```

## Root Cause

This is caused by the `slub_debug=P` option being added to the kernel cmdline parameters during boot. This configuration is required by [DISA STIG in Enterprise Linux 8](https://stigviewer.com/stigs/red_hat_enterprise_linux_8/2020-11-25/finding/V-230279)  and [DISA STIG in Enterprise Linux 9](https://stigviewer.com/stigs/red_hat_enterprise_linux_9/2024-12-04/finding/V-257794) to help mitigate [use-after-free based attacks](https://cwe.mitre.org/data/definitions/416.html).

The message does not indicate any issues with the server’s operation and can be safely ignored.

## References & related articles (Optional)

[DISA STIG Explanation](https://www.titania.com/resources/guides/disa-stig-compliance-explained)
[DISA STIG Requirements for Enterprise Linux 8](https://stigviewer.com/stigs/red_hat_enterprise_linux_8/)  
[DISA STIG Requirements for Enterprise Linux 9](https://stigviewer.com/stigs/red_hat_enterprise_linux_9/)  
[DISA STIG SLUB Requirements for Enterprise Linux 8](https://stigviewer.com/stigs/red_hat_enterprise_linux_8/2020-11-25/finding/V-230279)  
[DISA STIG SLUB Requirements for Enterprise Linux 9](https://stigviewer.com/stigs/red_hat_enterprise_linux_9/2024-12-04/finding/V-257794)  
[DISA STIG for Rocky Linux 8.x](https://docs.rockylinux.org/books/disa_stig/disa_stig_part1/)


Unhashed Kernel Memory Error in DMESG


## Introduction
[Infiniband network](https://www.nvidia.com/en-us/networking/products/infiniband/) needs a "TYPE" value for the network script. 

As discussed in [this thread](https://github.com/warewulf/warewulf/issues/457) The warewulf version 4.3, does not have a `TYPE=""` in the overlay: https://github.com/warewulf/warewulf/blob/v4.3.0/overlays/wwinit/etc/sysconfig/network-scripts/ifcfg.ww#L12

## Problem
The Infiniband network fails to come up. 

## Resolution
The fix was added in [version 4.4.0](https://github.com/warewulf/warewulf/releases/tag/v4.4.0). 
Upgrade Warewulf to a newer version.


## References & related articles
[Deploying a Dell PowerEdge and Cornelis Omni-Path Cluster with Warewulf](https://ciq.com/blog/deploying-a-dell-poweredge-and-cornelis-omni-path-cluster-with-warewulf/)
[Infiniband network](https://www.nvidia.com/en-us/networking/products/infiniband/)


Warewulf Config for Infinibad Interfaces


## Introduction

By default, Warewulf automatically fetches the latest kernel from the assigned container image and provides that kernel during PXE booting. Using the auto-detected kernel is recommended in the vast majority of cases. In addition, performing `dnf update` within the container should update to the latest kernel available.

## Problem

Some container images are created for specific Rocky Linux releases (e.g., Rocky Linux 8.8) and they would not have the right repositories to fetch updates, since they are built for that specific reason. Therefore the update process will not complete successfully.

## Symptoms

You may see the following when running `dnf update`:

```shell
# dnf update
Rocky Linux 8.8 - AppStream                                              4.6 MB/s |  12 MB     00:02
Rocky Linux 8.8 - BaseOS                                                 6.2 MB/s | 7.2 MB     00:01
Rocky Linux 8.8 - Extras                                                  24 kB/s |  14 kB     00:00
Dependencies resolved.
Nothing to do.
Complete!
```

## Resolution

There are few options available:

### 1 - Use a different image

Install a different image to make sure you can upgrade or downgrade to distinct versions. To change the container for your nodes without affecting operations, please follow [Best Practices for Applying OS Updates Across Node Images](https://kb.ciq.com/article/warewulf-node-os-update-best-practices).

### 2 - Use the `dnf upgrade` command

Use the `dnf upgrade` command instead and specify the release version you want to upgrade to:

```shell
$ dnf upgrade --releasever=8.10
Transaction Summary
=============================================================================================================================
Install   24 Packages
Upgrade  127 Packages

Total download size: 593 M
Is this ok [y/N]:
```

If you run into the following error while running the above command:

```shell
Errors during downloading metadata for repository 'appstream':
  - Status code: 404 for http://dl.rockylinux.org/vault/rocky/8.10/AppStream/x86_64/os/repodata/repomd.xml (IP: 151.101.54.132)
Error: Failed to download metadata for repo 'appstream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
```

Make sure to comment out the `_baseurl_` line in the `/etc/yum.repos.d` directory and uncomment the `_mirror_` line. You can use the command below:

```shell
sed -i -E 's/^(baseurl)/#\1/; s/^#(mirror)/\1/' /etc/yum.repos.d/*
```

Verify changes and try again.

### 3 - Change the repositories within the container image

You can manually set the `/etc/yum.repos.d/*` repositories to your preferred URLs for the version you want to upgrade to.

## References & related articles

[Warewulf Documentation - Images](https://warewulf.org/docs/v4.6.x/images/images.html)


Updating a Container's Kernel in Warewulf


## Introduction

Warewulf configures an external DHCP service to direct cluster nodes how to boot from Warewulf. In its default configuration, Warewulf uses a DHCP pool to provision cluster nodes, after which they transition to the permanent IP address configured in the node registry.

---
**_⚠️ WARNING_** _The DHCP range must not overlap with the addresses that are assigned to nodes in the node registry._

---

To define the desired range of IP addresses to lease during the boot process, update `dhcp.range start` and `dhcp.range end`.

```yaml
dhcp:
  range start: 10.0.0.100
  range end: 10.0.0.110
```

---
**_NOTE:_** _The DHCP range is only used while a node is provisioning, and its size determines how many nodes can provision simultaneously. For example, a large cluster of thousands of nodes may be provisioned in batches of 32 nodes, so a DHCP pool of 32 addresses should be sufficient._

---

Warewulf may alternatively also be configured to provide static DHCP addresses based on individual node configurations. This requires additional administrative overhead, as the DHCP configuration must be updated when nodes are added to the cluster. It removes the need for a DHCP address pool to be separate from the assigned cluster node IP addresses.

```yaml
# warewulf.conf

dhcp:
  template: static
```

The Warewulf daemon should be restarted after the changes made to `warewulf.conf`, and the DHCP service must be reconfigured to generate the necessary static DHCP host entries.

``` bash
$ systemctl restart warewulfd

$ wwctl configure dhcp
Building overlay for wwctl1: host
Enabling and restarting the DHCP services

$ grep --after-context=5 '^host n1-default' /etc/dhcp/dhcpd.conf
host n1-default
{
    hardware ethernet 5a:00:05:23:eb:c6;
    fixed-address 10.0.0.111;
    option host-name "n1";
}
```

## Problem

Some users may try to change the `dhcp.template` option to `static` instead of the default config. As mentioned, it will require extra management to configure it the first time. This may cause the nodes to be unable to boot, as it may miss required information that has to be manually added in the `/etc/dhcpd.conf` file.

## Resolution

The DHCP configuration is only for booting the node. The best practice is to set the `dhcp.template:` to `default` and let the node boot with `autodiscovery`. Once it boots, it will get the IP address that was set on the node configuration. When this happens, Warewulf will save the MAC address in the `node.conf` file, and it will be easier to switch `dhcp.template:` to `static` in the future.

## References & related articles

[Warewulf v4.5.x Control Server Setup Documentation](https://warewulf.org/docs/v4.5.x/contents/setup.html)


DHCP Not Discovering Nodes if Set as Static


## Introduction

Kdump utilizes `kexec` to swiftly boot into a dump-capture kernel whenever a memory dump of the system kernel is required, such as during a system panic. This process preserves the system kernel's memory image across the reboot, making it accessible to the dump-capture kernel.

You will learn to how to enable `kdump` on one of your Compute Nodes in your Warewulf cluster. This is essential for when a kernel panic occurs and for performing an in-depth analysis at the kernel level.

## Prerequisites

_This guide assumes that a Warewulf server is successfuly installed and nodes are able to boot from a Rocky Linux image._

In many cases, the `kdump` service is installed and activated by default on new Linux installations, but some installation options do not have to install or enable `kdump` by default.

If you do not know whether `kdump` is installed on your system, you can "_shell into_" an image with `wwctl image shell` like the example below. For Warewulf 4.5.x versions, you can use `wwctl container shell`.

You can "_shell into_" the image with `wwctl image shell`. For 4.5.x versions you can use `wwctl container shell`.

```shell
[root@Warewulf ~]# wwctl image shell rockylinux-9
Image build will be skipped if the shell ends with a non-zero exit code.
```

From there you can check if the RPM has been installed by running `rpm -q kexec-tools`:

```shell
[warewulf:rockylinux-9] /# rpm -q kexec-tools
package kexec-tools is not installed
```

If you need to install `kexec-tools`, please run the command `dnf install kexec-tools`. This command secures installation of the userspace tools for `kexec`:

```shell
[warewulf:rockylinux-9] /# dnf install kexec-tools

Last metadata expiration check: 0:01:12 ago on Wed Mar  5 18:07:55 2025.
Dependencies resolved.
=============================================================================================================================================================================================
 Package                                        Architecture                           Version                                                  Repository                              Size
=============================================================================================================================================================================================
Installing:
 kexec-tools                                    x86_64                                 2.0.27-16.el9_5.1                                        baseos                                 477 k

...

Complete!
```

## Installation instructions

### Configure kdump

The Linux kernel accepts several boot-time arguments. For example, Warewulf currently specifies the following arguments by default:
`quiet crashkernel=no vga=791 net.naming-scheme=v238`.

You can specify a different set of kernel arguments for a node or profile using `--kernelargs`. This value is recorded on a node or profile as the `Kernel.Args` field.

**To set this at the profile level, run this command:**

```bash
wwctl profile set default --kernelargs 'quiet,crashkernel=512M,vga=791,net.naming-scheme=v238'
```

**Similarly, to set this at the node level, run this command:**

```bash
wwctl node set node01 --kernelargs 'quiet,crashkernel=512M,vga=791,net.naming-scheme=v238'
```

_To specify the memory reserved for the `kdump` kernel, set the `crashkernel=` option to the required memory value. For example, to reserve 128 MB of memory, you can use `crashkernel=128M`._

---
**_NOTE_:** _For newest Warewulf versions, `kernel args` is set as a list to be able to combine them between profiles and nodes, therefore, these values may get appended. You potentially have to negate default values by adding "`~`". An example is: `~crashkernel=no`._

---

### Changing the `excludes` file inside the container to allow copying the `/boot` directory to the nodes

Warewulf can exclude files from an image to prevent them from being delivered to the compute node. This is typically used to reduce the size of the image when some files are unnecessary. Patterns for excluded files are read from the file `/etc/warewulf/excludes` in the image itself. For example, the default Rocky Linux images exclude these paths:

```shell
/boot/
/usr/share/GeoIP
```

To remove `/boot/` from the `excludes` file, shell into the image:

```shell
[root@Warewulf ~]# wwctl image shell rockylinux-9
Image build will be skipped if the shell ends with a non-zero exit code.
[warewulf:rockylinux-9] /# sed -i '/\/boot\//d' /etc/warewulf/excludes
[warewulf:rockylinux-9] /# cat /etc/warewulf/excludes
/usr/share/GeoIP
```

Since `kdump` needs the `vmlinuz` file to boot the dump kernel, you need to add a symlink to `/usr/lib/modules/<kernel_version>/<vmlinuz file>`, as this file is not present on `/boot`:

```shell
[warewulf:rockylinux-9] /# KERNEL_VERSION=$(ls /usr/lib/modules | head -n 1)
ln -s /usr/lib/modules/$KERNEL_VERSION/vmlinuz-$KERNEL_VERSION /boot/vmlinuz-$KERNEL_VERSION
[warewulf:rockylinux-9] /# ls /boot/ | grep vmlinuz
vmlinuz-5.14.0-503.19.1.el9_5.x86_64
```

### Configuring the kdump target

When a kernel crash is captured, the core dump can be stored in various ways: as a file in a local file system, directly on a device, or sent over a network using the NFS (Network File System) or SSH (Secure Shell) protocols. Only one of the three options can be configured at a time. By default, the `vmcore` file is stored in the `/var/crash` directory of the local file system. To verify this, check the `/etc/kdump.conf` file's contents. In this example, NFS is used, since the Warewulf nodes will clear anything in memory after a reboot, and anything added to `/var/crash` will be removed.

An example from the `/etc/kdump.conf` file:

```shell
# nfs <nfs mount>
#           - Will mount nfs to <mnt>, and copy /proc/vmcore to
#             <mnt>/<path>/%HOST-%DATE/, supports DNS.
```

#### Configure `kdump` manually on the container

_Shell into_ the image and run the following commands:

```shell
[warewulf:rockylinux-9] /# grep -v ^# /etc/kdump.conf

auto_reset_crashkernel yes
path /var/crash
core_collector makedumpfile -l --message-level 7 -d 31

[warewulf:rockylinux-9] /# vi /etc/kdump.conf

# Edit the fields accordingly 
# If adding nfs please comment path

auto_reset_crashkernel yes
#path /var/crash
nfs nfs_server_ip:/kdump/path
core_collector makedumpfile -l --message-level 7 -d 31
```

#### Configuring kdump using overlays

To create the `kdump` overlay, please run:

```shell
$ wwctl overlay create kdump
```

Edit the `kdump.conf.ww` file (this command will create the overlay template):

```shell
$ wwctl overlay edit --parents kdump /etc/kdump.conf.ww
```

Add the following configuration after the `# This file is autogenerated by warewulf` line:

```bash
auto_reset_crashkernel yes
core_collector makedumpfile -l --message-level 7 -d 31

{{ if .Tags.kdump_nfs_location }}
nfs {{ .Tags.kdump_nfs_location }}
{{ else }}
path /var/crash
{{ end }}
```

Check that the new changes were applied successfully:

```shell
wwctl overlay show kdump /etc/kdump.conf.ww
```

Display the changes on a node to confirm it is working as expected:

```shell
$ wwctl overlay show -r node01 kdump /etc/kdump.conf

backupFile: true
writeFile: true
Filename: /etc/kdump.conf
# This is a Warewulf Template file.
#
# This file (suffix '.ww') will be automatically rewritten without the suffix
# when the overlay is rendered for the individual nodes. Here are some examples
# of macros and logic which can be used within this file:
#
# Node FQDN = node01
# Node Cluster = oso
# Network Config = <no value>, <no value>, etc.
#
# Go to the documentation pages for more information:
# https://warewulf.org/docs/main/contents/overlays.html
#
# Keep the following for better reference:
# ---
# This file is autogenerated by warewulf

auto_reset_crashkernel yes
core_collector makedumpfile -l --message-level 7 -d 31


path /var/crash
```

Change the tag to add the NFS path by using the `--tagadd` flag. In the following example, the path used was `172.16.131.5:/var/crash/`:

```shell
$ wwctl profile set default --tagadd kdump_nfs_location=172.16.131.5:/var/crash/
Are you sure you want to modify 1 profile(s): y
```

Render the changes on a node to confirm the changes:

```shell
$ wwctl overlay show -r node01 kdump /etc/kdump.conf
# Removing the extra lines to shorten the output
...

nfs 172.16.131.5:/var/crash/
```

Add the newly created overlay to the default profile, and boot the node:

```shell
wwctl profile set default -O $(wwctl profile list default -a | grep SystemOverlay | awk '{print $3,kdump}')
```

### Testing the kdump configuration

---
**_⚠️ WARNING_** _The commands below cause the kernel to crash. Use caution when following these steps, and by no means run them on a production system._

---

To test the configuration, `ssh` into the node, and make sure that the service is running:

```shell
$ systemctl status kdump.service 

#If the service is active, please run the following commands:

$ echo 1 > /proc/sys/kernel/sysrq
$ echo c > /proc/sysrq-trigger
```

The above commands force the Linux kernel to crash, and the `address-YYYY-MM-DD-HH:MM:SS/vmcore` file is copied to the location you have selected in the configuration (that is, to `/var/crash/` by default).

## References & related articles

[Kdump docs](https://www.kernel.org/doc/Documentation/kdump/kdump.txt)  
[kdump.conf man page](https://linux.die.net/man/5/kdump.conf)  
[Warewulf documentation](https://warewulf.org/docs/)  


How to Enable Kdump in Warewulf Nodes


## Introduction

This article outlines the best practices for applying operating system (OS) updates across node images in a Warewulf environment. Following these practices helps ensure the stability and reliability of your system during updates.

## Problem

Administrators often face challenges when applying OS updates across node images, including the risk of breaking the image or having difficulties in reverting changes. Ensuring a smooth update process is crucial for maintaining the operational integrity of the system.

## Symptoms

- Inconsistent node behavior post-update.
- Node failures or inability to boot.
- Difficulty in reverting to a previous stable state after updates.

## Resolution

To minimize risks and ensure a smooth update process, follow these best practices in order of preference:

### Best practices when updating images

When planning OS updates in a Warewulf environment, here are three approaches you may consider. The primary method builds a new image using an updated Containerfile which can be run through to a CI/CD pipeline, the secondary copies the current image to a new image for updating, and the final method directly updates the existing image, which is riskier as it will not retain the original, working image.

#### Primary

1. **Update the Containerfile**: Apply updates to the Containerfile used to build the node image.
2. **Build a New Image**: Construct a new image from the updated Containerfile.
3. **Import into Warewulf**: Import the newly built image into Warewulf.
4. **Gradual Migration**: Gradually migrate nodes to the new image to monitor and ensure stability.

#### Secondary

1. **Copy Current Image**: Duplicate the current node image and assign a new name to the copy.
2. **Apply Updates**: Implement the necessary updates on the copied image.
3. **Gradual Migration**: Gradually transition nodes to the updated image to ensure stability and functionality.

#### Tertiary

1. **Direct Updates**: Apply updates directly to the current node image. This method is riskier and should only be used if the first two methods are not feasible.
2. **Backup**: Ensure there is a backup of the current image before proceeding with updates. This allows for a rollback if the update process fails or causes issues.

### Adding image to nodes

When creating a new container, you have a couple options for applying it within your environment, primarily at either the profile level or the node level. Configuration set directly on a node overrides configuration from a profile. For example, we can configure a node to use a different container image than what's configured on the default profile.

#### Updating individual nodes

In this example, we will select a specific test node, **n8**, to upgrade to newly built 8.10 Rocky Linux image. Please note that in Warewulf `4.5.8` and prior, the flag to add or edit a container image was `--container`. In `4.6.0` this was changed to `--image` to better reflect its purpose. However, the `--container` flag still works for compatibility.

```bash
$ wwctl node set n8 --container=rockylinux-8.10 

$ wwctl node list n8 --all
  NODE  FIELD                     PROFILE     VALUE
  n8    Id                        --          n8
  n8    Comment                   default     This profile is automatically included for each node
  n8    ImageName                 SUPERSEDED  rockylinux-8.10
  n8    Ipxe                      --          (default)
  n8    RuntimeOverlay            --          (generic)
  n8    SystemOverlay             --          (wwinit, [...])
  n8    Root                      --          (initramfs)
  n8    Init                      --          (/sbin/init)
  n8    Kernel.Args               --          (quiet crashkernel=no vga=791 net.naming-scheme=v238)
  n8    Profiles                  --          default,cluster1
  n8    PrimaryNetDev             --          (default)
  n8    NetDevs[default].Type     --          (ethernet)
  n8    NetDevs[default].OnBoot   --          (true)
  n8    NetDevs[default].Device   cluster1    eno1
  n8    NetDevs[default].Netmask  cluster1    255.255.255.0
  n8    NetDevs[default].Gateway  cluster1    10.0.0.3
  n8    NetDevs[default].Primary  --          (true)
```

The **ImageName** field for the n8 node is now set to `rockylinux-8.10`. The **SUPERSEDED** value in the profile column indicates that a node value is replacing a profile value typically set from a profile. This allows you to test the new image on an individual server before rolling it out to a wider selection of nodes.

#### Updating a profile

We can also create a new profile, in our example we will call it **dev**, to roll this out to multiple nodes at once. This is useful, particularly when testing, when you have multiple generations of hardware or configurations of hardware that need to be tested and validated to work with the new image. We will first start by creating the profile if it has not been created already.

```shell
$ wwctl profile add dev --comment="Development/testing environment"

$ wwctl profile list
  PROFILE NAME  COMMENT/DESCRIPTION
  dev           Development/testing environment
  default       This profile is automatically included for each node
```

Once this new `dev` profile is set, we can add information like we would on a node, In our case, lets apply a newer `rockylinux-8.10v2` image to this profile.

```shell
wwctl profile set dev --container=rockylinux-8.10v2
```

We can now add our node `n8` to this profile

```shell
wwctl node set n8 --profile=default,dev
```

While you can configure a node with one profile, one of the most powerful features of Warewulf is the ability to combine profiles. If two profiles set the same field such as the image field, the right-most profile in the node’s list takes precedence. In our example above, the image from `dev` will take precedence over the image in `default`. Reminder, values set directly on nodes will take precedence over profile field values.
If you follow this guide, `n8` was set to use `rockylinux-8.10` as its image directly and as a result, this value overrides anything within the profiles. If we unset this value on the node, we will see `n8` use the `rockylinux-8.10v2` image:

```shell
wwctl node set n8 --container=UNSET
```

```shell
$ wwctl node list n8 -a
NODE  FIELD                        PROFILE  VALUE
----  -----                        -------  -----
n8    Profiles                     --       default,dev
n8    Comment                      dev      Development/testing environment
n8    ImageName                    dev      rockylinux-8.10v2
[...]
```

## Root Cause

The risk of breaking the image or encountering issues increases significantly when updates are applied directly to the current image. Using a Containerfile to build and update images or duplicating the current image before applying updates helps maintain stability and provides a clear rollback path.

## Notes

- Always ensure there is a backup of the current image before applying any updates.
- Gradual migration allows for monitoring and quick identification of issues, reducing the impact on the overall system.

## References & related articles

[Warewulf Profile Set Documentation](https://warewulf.org/docs/v4.6.x/reference/wwctl_profile_set.html)  
[Warewulf Node Set Documentation](https://warewulf.org/docs/v4.6.x/reference/wwctl_node_set.html)  
[Warewulf Profiles Documentation](https://warewulf.org/docs/v4.6.x/nodes/profiles.html)  
[Warewulf Images Documentation](https://warewulf.org/docs/v4.6.x/images/images.html)


Best Practices for Applying OS Updates Across Node Images


## Introduction

A `tmpfs` filesystem is a temporary, memory-based filesystem that resides in virtual memory, providing extremely fast file access. It dynamically allocates RAM and can use `swap` space when needed, consuming only as much memory as required for its contents. However, its contents are volatile, meaning all data is lost when the filesystem is unmounted. See [tmpfs(5)](https://man7.org/linux/man-pages/man5/tmpfs.5.html) for more details.

## Problem

The user doesn't want `tmpfs` to use `swap` space. For example, swap space set aside for slurm jobs.

## Resolution

As mentioned in the [warewulf documentation](https://warewulf.org/docs/main/contents/boot-management.html), the `wwinit` module provisions to `tmpfs`. By default, `tmpfs` is permitted to use up to 50% of physical memory. This size limit may be adjusted using the kernel argument `wwinit.tmpfs.size`. (This parameter is passed to the size option during `tmpfs` mount.)

`wwinit.tmpfs.size` is only used by [dracut](https://man7.org/linux/man-pages/man8/dracut.8.html). To boot with dracut please follow [this guide.](https://warewulf.org/docs/main/contents/boot-management.html#booting-with-dracut).

If you aren’t using dracut this parameter won’t adjust the `tmpfs` limit, and is not currently configurable, aside from manually editing the [`init` file](https://github.com/warewulf/warewulf/blob/main/overlays/wwinit/rootfs/init#L32) in the overlay, or maintaining a copy of it locally.

`tmpfs` blocks may be swapped out, when there is a shortage of memory. `tmpfs` has a mount option to disable its use of `swap`:

``` bash
noswap - # Disables swap. Remounts must respect the original settings. By default swap is enabled.
```

This can also be managed in the [init file](https://github.com/warewulf/warewulf/blob/main/overlays/wwinit/rootfs/init#L32)

## Notes

Kernel args can still be set with dracut boot. However, `--kerneloverride` as currently implemented would require the ability to push the matching kernel modules into the dracut image, which is not currently supported.

## References & related articles

[Warewulf - Booting with Dracut](https://warewulf.org/docs/main/contents/boot-management.html#booting-with-dracut)  
[Man tmpfs page](https://man7.org/linux/man-pages/man5/tmpfs.5.html)  
[Man dracut page](https://man7.org/linux/man-pages/man8/dracut.8.html)


How to Limit Tmpfs for Taking Swap Space


## Introduction

In a typical Warewulf environment, compute nodes run `wwclient` to pull runtime overlays from the head node. A node may fail to connect to the provisioning server after boot, resulting in a client timeout. This article details a step-by-step troubleshooting process to verify connectivity, inspect configuration settings, and adjust the Warewulf client configuration if needed.

## Problem

When there is a connection issue, you might observe errors such as:

```text
Job for wwclient.service failed because a timeout was exceeded
```

```text
Failed to start Warewulf node runtime overlay update
```

## Resolution

### Verify connectivity and server configuration

A timeout error generally indicates that the compute node is unable to reach the provisioning server. Specifically, it is not able to download the runtime file over the designated Warewulf port. By default, this port is typically set to **9873**. Follow these steps to verify connectivity:

#### Check the client configuration

Confirm the IP address and port that the `wwclient` is set to use. For example, review the configuration file and verify the settings from a booted compute node:

```bash
cat /warewulf/warewulf.conf | grep -E "ipaddr:|port:"
```

*Expected output (with your server’s IP address and port):*

```text
ipaddr: <SERVER_IP>
port: 9873
```

#### Test server connection

Use `curl` to request the provision directory and verify that the server is reachable:

```bash
curl -I http://<SERVER_IP>:9873/provision/
```

*A successful response should look like:*

```text
HTTP/1.1 200 OK
Content-Length: 3779
Content-Type: text
Date: [Date and Time]
```

If you receive an error such as `curl: (7) Failed to connect to <SERVER_IP>`, then it is likely that a firewall rule on the server, or between the node and server, is blocking access. To resolve firewall issues on a system using `firewalld`, run:

```bash
firewall-cmd --permanent --add-service=warewulf
firewall-cmd --reload
```

#### Test the runtime file URL

The `wwclient` downloads a runtime file using a URL with several parameters. The URL format is:

<http://{ipaddr}:{port}/provision/{wwid}?assetkey={tag}&uuid={localUUID}&stage=runtime&compress=gz>

- **WWID:** Usually the node’s MAC address, which you can verify from the boot command line:

  ```bash
  cat /proc/cmdline | grep -oP 'wwid=[^ ]+'
  ```

- **Asset Key and UUID:** Values are from `dmidecode`. If `dmidecode` is not available or the values are empty, the key and uuid will default to "*Unknown*."

Test the download with the url you generate. For example:

```bash
curl -I "http://192.168.1.2:9873/provision/aa:bb:cc:dd:ee:ff?assetkey=Unknown&uuid=Unknown&stage=runtime&compress=gz"
```

*A correct response will indicate an `HTTP 200 OK` status and appropriate content headers. If you can download the file or the headers indicate success, the connection is established correctly.*

### Create a custom warewulf configuration for the wwclient

If the provisioning network is unavailable after boot and you need a custom configuration with a different IP address, you can create a new overlay for the `wwclient`. By default, the client reads its configuration from `/warewulf/warewulf.conf`, which is copied from `/etc/warewulf/warewulf.conf` on the host server. An [issue](https://github.com/warewulf/warewulf/issues/1788) has been submitted for the ip address to be configured at the host or profile level. Until then, and for Warewulf `4.6.0` and below, proceed with the following steps to create a customized overlay:

#### Create a new overlay

Create an overlay called `custom-wwclient`:

```bash
wwctl overlay create custom-wwclient
```

#### Import and edit the Warewulf configuration file

Copy the existing configuration file into the new overlay and open it for editing. For example, to copy it to a custom location such as `/warewulf/custom-warewulf.conf`:

```bash
wwctl overlay import -p custom-wwclient /etc/warewulf/warewulf.conf /warewulf/custom-warewulf.conf
wwctl overlay edit custom-wwclient /warewulf/custom-warewulf.conf
```

In your editor, modify the `ipaddr` field to the appropriate IP Address. You can also change the port here if desired.

#### Modify the systemd unit file

Next, adjust the systemd unit file so that the `wwclient` uses the new configuration file. Edit the `wwclient` service as follows:

```bash
wwctl overlay edit -p custom-wwclient /etc/systemd/system/wwclient.service
```

Update or add the following:

```ini
[Unit]
Description=Warewulf Node Runtime Overlay Update
After=network.target local-fs.target

[Service]
Type=notify
ExecStart=/warewulf/wwclient --warewulfconf /warewulf/custom-warewulf.conf
ExecReload=/bin/kill -s SIGHUP "$MAINPID"
PIDFile=/var/run/wwclient.pid
TimeoutSec=60

[Install]
WantedBy=multi-user.target
```

#### Link the overlay with a profile and build

Add the new overlay to the appropriate profile, in our case **default**, ensuring to keep any existing overlays. We will be adding this as a system overlay so it is applied at boot.

```bash
wwctl profile set default --wwinit=wwinit,custom-wwclient
```

Then, build the new overlay settings:

```bash
wwctl overlay build
```

#### Reboot and verify

Reboot the node to ensure it picks up the new configuration. Once up, verify that the process is running with the adjusted configuration file:

```bash
ps aux | grep wwclient
```

You should see a command line similar to:

```text
/warewulf/wwclient --warewulfconf /warewulf/custom-warewulf.conf
```

## References & Related Articles

[Rocky Linux Firewalld Guide](https://docs.rockylinux.org/guides/security/firewalld-beginners/)  
[Warewulf Documentation](https://warewulf.org/docs/v4.5.x/)  
[Warewulf Overlays](https://warewulf.org/docs/v4.5.x/contents/overlays.html)  


Troubleshooting Warewulf Client Timeout Issues


## Introduction

When attempting to install Rocky Linux over a slow or unreliable network connection including through IPMI (Intelligent Platform Management Interface), you may encounter timeout issues that prevent the installer from loading properly.

## Problem

When the installer fails to load within the expected time frame, you may encounter the following error message:

```text
X did not start in the expected time, falling back to text mode.
```

This error occurs because the graphical installer (X server) is unable to start within the default timeout period. This is often caused by slow network speeds, high latency, or interruptions in the connection, which delay the loading of necessary resources. This issue can also occur when a server is configured to use 802.3ad (LACP) for link aggregation but is not configured to support this during the iPXE boot process. As a result, the network interfaces may fail to establish a proper connection, leading to network communication problems. As a result, the installer fails over to text mode, which may not be desired.

## Resolution

If switching to a faster network connection is not an option, there are two primary methods to resolve this issue:

1. **Increase the timeout period** to give the installer more time to load.
2. **Load the installation media into memory**, which bypasses the need to continuously fetch data over the network.

Both methods involve editing the kernel parameters during the installation process.

### Increase timeout

The default timeout for the installer is 60 seconds. You can increase this timeout by modifying the kernel parameters.

1. During the boot process, press `e` to edit the kernel boot options.
2. Locate the line that begins with `linux` or `linuxefi` and contains the kernel parameters.
3. Add the following parameter to the end of the line:

   ```text
   inst.xtimeout=<SECONDS>
   ```

   For example, to set the timeout to 5 minutes (300 seconds), you would add:

   ```text
   inst.xtimeout=300
   ```

4. Press `Ctrl-X` or `F10` to continue booting with the new parameter.

The installer will now wait up to the specified timeout period before falling back to text mode.

### Load into memory

If increasing the timeout does not resolve the issue or if the network connection is highly unstable, you can force the installer to load the entire installation media into memory. This eliminates the need to continuously fetch data over the network during the installation process.

1. As with the previous method, press `e` to edit the kernel boot options.
2. Add the following parameter to the end of the line:

   ```text
   rd.live.ram
   ```

   This option instructs the installer to copy the entire ISO or installation image into memory.

3. Continue with the installation.

Depending on the size of the installation media and the speed of the network, this process may take some time and you will not see any progress or status update.

## Notes

Loading the installation media into memory can require significant RAM. The amount of memory needed depends on the size of the ISO or image and the packages included. For example, a minimal Rocky Linux installation image may require only a few gigabytes of RAM, while a full *DVD* image with additional packages could require significantly more.

If you’re working in a slow or unreliable network environment and need to install or manage multiple servers, it may be worth setting up a [local mirror](https://kb.ciq.com/article/create-local-repo-using-reposync). A local mirror allows you to host the necessary installation files and package updates within your network, significantly reducing dependency on external connections.

## References & related articles

[How to Create a Local Mirror Using Reposync with CIQ Depot](https://kb.ciq.com/article/create-local-repo-using-reposync)  


CIQ Knowledgebase

By category

By topic