
## Introduction

In environments that use user namespaces, administrators must keep user and group ID mappings consistent across multiple hosts to maintain proper access control. Many try to achieve this by symlinking `/etc/subuid` and `/etc/subgid` to shared storage, which distributes the same mappings everywhere.

However, [Apptainer](https://docs.ciq.com/apptainer/) and similar tools rely on the `newuidmap` and `newgidmap` utilities, which refuse to read symbolic links. When you use symlinks for these files, Apptainer’s fakeroot builds fail. This article explains why this happens and shows safer ways to synchronize mappings across hosts.

## Problem

If you symlink `/etc/subuid` and `/etc/subgid` to shared storage, Apptainer’s fakeroot feature breaks. The `newuidmap` and `newgidmap` utilities reject symlinked files to prevent privilege escalation and ensure file integrity. Because these tools can’t read the mappings, any fakeroot build or container that depends on them fails.

This incompatibility prevents Apptainer from working correctly in environments that distribute user mappings through symbolic links.

## Symptoms

When you build an Apptainer image in fakeroot mode on a system with symlinked `subuid` or `subgid` files, you might see errors like:

```bash
apptainer build --fakeroot rockylinux_fakeroot.sif docker://rockylinux:9
ERROR  : 'newgidmap' execution failed. Check that 'newgidmap' is setuid root or has setcap cap_setgid+eip.
ERROR  : Error while waiting event for user namespace mappings: no event received
```

These messages show that Apptainer can't create the required user namespace mappings because `newuidmap` and `newgidmap` fail to open the symlinked configuration files.

## Root cause

`newuidmap` and `newgidmap` open the `/etc/subuid` and `/etc/subgid` files with the `O_NOFOLLOW` flag. This flag tells the kernel not to follow symbolic links. When the path ends in a symlink, the system call fails with an `ELOOP` error "Too many levels of symbolic links".

For example, an `strace` trace shows the failure:

```bash
openat(AT_FDCWD, "/etc/subgid", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW) = -1 ELOOP (Too many levels of symbolic links)
```

Because of this behavior, the utilities can't access symlinked mapping files, and Apptainer fails to set up user namespaces when using fakeroot mode.

## Resolution

To synchronize `subuid` and `subgid` files across hosts without using symlinks, consider alternative methods such as:

### `Rsync` and `Cron` jobs

Use `rsync` in conjunction with a `cron` job to periodically sync these files across hosts. For details, refer to the [Rocky Linux Rsync Guide](https://docs.rockylinux.org/guides/backup/rsync_ssh/) and [Rocky Linux Cron Guide](https://docs.rockylinux.org/guides/automation/cron_jobs_howto/).

### Ascender/Ansible automation

Automate file synchronization with Ansible for a more flexible and scalable solution. Ascender, an orchestration tool from CIQ, can assist in managing these automation tasks. For more details, refer to the [Rocky Linux Ansible Guide](https://docs.rockylinux.org/books/learning_ansible/02-advanced/) and [Ascender Automation](https://ciq.com/products/ascender/).

## References & related articles

[Rocky Linux Rsync Guide](https://docs.rockylinux.org/guides/backup/rsync_ssh/)  
[Rocky Linux Cron Guide](https://docs.rockylinux.org/guides/automation/cron_jobs_howto/)  
[Rocky Linux Ansible Guide](https://docs.rockylinux.org/books/learning_ansible/02-advanced/)  
[Ascender Pro](https://ciq.com/products/ascender/)


Error When Symlinking User and Group ID Mapping


## Introduction

This document outlines how to add an SSL certificate to an existing Ascender installation, assuming you have a valid and working Ascender installation. The recommended installation method is the [ascender-install](https://github.com/ctrliq/ascender-install) script from the CIQ repository.

A requirement for successfully adding an SSL certificate is to have one initially generated. This is not discussed in this article and creating an SSL certificate is necessary before continuing with this guide. The Common Name (CN) field in the certificate needs to exactly correlate with the URL you have set for your Ascender instance. Both the certificate and private key must be in `PEM` format. The certificate has to be stored with a `.crt` or `.cer` file extension and the private key assigned a `.key` file extension.

## Problem

When Ascender was installed, the `http` option was chosen instead of `https`. As a result, Ascender is using a default self-signed certificate and has no options within the `custom.config.yml` file to use other certificates.

## Resolution

After installing Ascender with the [ascender-install](https://github.com/ctrliq/ascender-install) scripts from CIQ, a `custom.config.yml` file is created. This includes all the variables required for the initial install process. Change each of the variables in this file like the example below:

* **kube_install**: In your case, a kubernetes cluster on Ascender is running already. This value needs to be set to `false`.
* **download_kubeconfig**: Similarly to the above, this value also needs to be configured as `false`. This is because a valid `KUBECONFIG` file used for authentication already exists in your Ascender instance.
* **k8s_lb_protocol**: Change this from `http` to `https`.

Go to the bottom of the file and include these lines:

* **tls_crt_path**: Details the path of where the certificate is located.
* **tls_key_path**: Here will be the path to where the private key is found.

An example of the above configuration steps:

```yaml
tls_crt_path: ~/ascender.crt
tls_key_path: ~/ascender.key
```

Run the `setup.sh` script again and you will find the new certificate successfully installed. Run the command illustrated below to observe if the new certificate has been successfully applied or not:

```bash
echo | openssl s_client -connect your-ascender-hostname:443 2>/dev/null | openssl x509 -noout -subject -ext subjectAltName -dates
```

## References & related articles

[CIQ Ascender](https://ciq.com/products/ascender)  
[Ascender documentation](https://docs.ciq.com/ascender/)  
[Ascender Install Scripts and Documentation](https://github.com/ctrliq/ascender-install)  
[Ascender Feature Overview](https://ciq.com/blog/manage-enterprise-infrastructure-with-ascender-by-ciq-what-is-ascender/)  
[Ascender Installation Video](https://www.youtube.com/watch?v=j15pFNfs8dw)  
[Generating SSL Keys](https://docs.rockylinux.org/guides/security/ssl_keys_https/)  
[Generating SSL Keys with Let's Encrypt](https://docs.rockylinux.org/guides/security/generating_ssl_keys_lets_encrypt/)  


Add an SSL Certificate to an Ascender Instance That Was Set Up Without a Certificate


## Introduction

This guide will have you setup and run Ascender on a K3s cluster.

## Problem

Many customers face difficulties when managing their infrastructure at scale. This is where [Ascender](https://ciq.com/products/ascender/) comes in to ensure scalable patching, security hardening and more across your fleet.

## Symptoms

A new user may encounter difficulties installing Ascender onto their K3s cluster.

## Resolution

### Prerequisites

* Same OS family as Rocky Linux. Rocky Linux 8.x or Rocky Linux 9.x is recommended as the base OS.

* System requirements:

  * CPUs: 2

  * Memory: 8GB (if installing both Ascender and Ledger)

  * Disk: 20GB (if installing both Ascender and Ledger)

* Internet access for the K3s cluster.

* Perform the below steps with a user that has privilege escalation. Do not use the `root` user account.

### Setup

* Update the node running K3s:

```bash
sudo dnf update -y
```

* Install `git`:

```bash
sudo dnf install -y git
```

* Allow these ports in `firewalld`:

```bash
sudo firewall-cmd --permanent --zone=public --add-port=22/tcp
sudo firewall-cmd --permanent --zone=public --add-port=80/tcp
sudo firewall-cmd --permanent --zone=public --add-port=443/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5432/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5895/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5986/tcp
sudo firewall-cmd --reload
```

* Clone the Ascender repository:

```bash
git clone https://github.com/ctrliq/ascender-install.git
```

* Navigate to the installation directory:

```bash
cd ascender-install
```

* Run the `config_vars.sh` script:

```bash
./config_vars.sh
```

* The script will present you with multiple questions. Answer each of them according to how you want to set up your Ascender environment and if you already have K3s installed on your node.

* Run the setup script to install Ascender:

```bash
sudo ./setup.sh
```

* Once complete, the final playbook recap will look like the following example:

```bash
PLAY RECAP *************************************************************************************************************************
ascender_host              : ok=14   changed=6    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0
localhost                  : ok=72   changed=27   unreachable=0    failed=0    skipped=4    rescued=0    ignored=0

ASCENDER SUCCESSFULLY SETUP
```

### Accessing Ascender

* You can access Ascender directly via the internal CLUSTER IP address.

## Notes

* To monitor the Ascender setup on your K3s cluster during install:

```bash
kubectl get pod -n ascender -w
```

* For further details surrounding particular pods:

```bash
kubectl describe pod -n ascender [pod name]
```

## References & related articles

[Ascender General Prerequisites](https://github.com/ctrliq/ascender-install/tree/main?tab=readme-ov-file#general-prerequisites)
[Ascender K3s Install Guide](https://github.com/ctrliq/ascender-install/blob/main/docs/installation/k3s/README.md)


Quick Start Guide for Installing Ascender on K3s


## Introduction

When importing jobs or running playbooks in [Ascender](https://ciq.com/products/ascender/), users may encounter a warning message indicating a version mismatch between the AWX Galaxy Collection and the Ascender version.

## Problem

During job operations in Ascender, the following warning message is displayed:

```text
[WARNING]: You are running collection version 24.6.1 but connecting to AWX version 25.2.0.
```

This warning appears when the `awx.awx` Galaxy collection version is lower than the Ascender application version.

## Resolution

This warning message can be safely ignored and does not require any action. The warning indicates that the `awx.awx` Galaxy collection is pinned to a specific version (in this example, `24.6.1`), while the Ascender application itself is running a newer version (in this example, `25.2.0`). `24.6.1` is the latest release available for the `awx.awx` galaxy collection.

No action is required. The job import and execution will proceed normally despite this warning. The warning is informational only and does not indicate an error or compatibility issue.

## References & related articles

[Ascender Quick Start Guide](https://kb.ciq.com/article/ascender/asc-ascender-quick-start)
[Ascender General Prerequisites](https://github.com/ctrliq/ascender-install/tree/main?tab=readme-ov-file#general-prerequisites)


AWX Collection Version Warning During Job Run


## Introduction

This article covers a case where a job or inventory source in Ascender fails to find a collection that ships with the execution environment, because a custom `ANSIBLE_COLLECTIONS_PATH` value replaced the default collection paths instead of adding to them.

## Problem

Setting `ANSIBLE_COLLECTIONS_PATH` to a single project-relative path, for example `/opt/ansible/collections`, replaces the entire collections search path rather than extending it. Ansible then looks for collections only in that one directory. Collections bundled with the execution environment are no longer on the path, so their modules, plugins, and inventory plugins cannot be resolved even though they are present in the image.

## Symptoms

A task fails with a "couldn't resolve module/action" error, or an inventory source fails with an "unknown plugin" error, for a collection you know is installed in the execution environment.

The `ansible-inventory` or `ansible-config` diagnostic output shows a collection location that includes only your custom path, for example:

```text
ansible collection location = /runner/project/collections
```

## Resolution

Append the default collection paths to your custom value rather than replacing them. Under **Settings > Jobs Settings > Extra Environment Variables**, set:

```json
{
  "ANSIBLE_COLLECTIONS_PATH": "/opt/ansible/collections:/runner/.ansible/collections:/usr/share/ansible/collections"
}
```

The first entry keeps your own collections directory, and the two default entries restore access to the collections bundled with the execution environment. After saving, re-run the job or inventory sync and the collection resolves correctly.

## Notes

`ANSIBLE_COLLECTIONS_PATH` is a full replacement, not an addition. Any time you set it, include every path Ansible needs to search, not only your own. The same rule applies to `ANSIBLE_ROLES_PATH` and similar path variables.

## References & related articles

[Building a Custom Execution Environment for Ascender](/article/ascender/asc-custom-execution-environment)  
[Update the Ascender Execution Environment on Kubernetes](/article/ascender/asc-update-execution-environment)  
[Ascender Execution Environments](https://ascender-automation.org/userguide/execution_environments.html)  
[Ascender Execution Environment Setup Reference](https://ascender-automation.org/userguide/ee_reference.html)  
[Ansible collections search path](https://docs.ansible.com/ansible/latest/collections_guide/collections_installing.html)


Bundled Collection Not Found After Overriding ANSIBLE_COLLECTIONS_PATH


## Introduction

Ascender uses Execution Environments to run Ansible playbooks inside containerized environments. The default EE that ships with Ascender includes a standard set of Ansible collections, Python packages, and system binaries. However, some Ansible modules require additional Python libraries or system packages that aren't included in the default image.

This article covers how to build a custom Execution Environment that includes additional Python packages, system dependencies, or Ansible collections, and how to configure Ascender to use it.

## Problem

Certain Ansible modules require Python libraries that aren't included in Ascender's default Execution Environment. For example, the `netbox.netbox` collection modules require the `pynetbox` Python package. When running a playbook that uses one of these modules with `delegate_to: localhost` or as part of a dynamic inventory plugin, the task fails because the required Python library is not present in the EE container.

Unlike Ansible collections, which Ascender installs automatically during project sync via a `collections/requirements.yml` file, Python packages can't be installed at runtime. There isn't a `pip install` mechanism during job execution. The only way to include additional Python packages is to build a custom Execution Environment image that contains them.

## Symptoms

When a required Python package is missing from the Execution Environment, Ansible tasks fail with errors similar to:

```text
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (pynetbox) on localhost. Please read the module documentation and install it in the appropriate location."}
```

## Resolution

### Prerequisites

The following are required on your build host:

- `ansible-builder` (installed via pip)
- A container runtime: `podman` or `docker`
- Access to a container registry, such as GitHub Container Registry, Quay.io, or a private registry

### Step 1: Clone the Ascender EE repository

CIQ publishes the source for the default Ascender Execution Environment. Fork or clone this repository as your starting point:

```bash
git clone https://github.com/ctrliq/ascender-ee
cd ascender-ee
```

### Step 2: Modify the EE definition

Edit `execution-environment.yml` to add your required dependencies.

To add Python packages, add entries to the `python:` block:

```yaml
python: |
  pynetbox
  ansible-sign
  jmespath
```

To add system packages, add entries to the `system:` block in the same file.

To add Ansible collections, add entries to the `galaxy:` block.

### Step 3: Install ansible-builder

If you don't already have `ansible-builder` installed on your build host:

```bash
pip3 install ansible-builder
```

### Step 4: Build the custom EE image

Build the image, tagging it for your target registry:

```bash
ansible-builder build -t your-registry.example.com/custom-ascender-ee:v1 -v3
```

The first build pulls in all collections and system packages, so it takes some time. Subsequent builds are faster due to layer caching.

### Step 5: Push the image to your registry

```bash
podman push your-registry.example.com/custom-ascender-ee:v1
```

If you are using the `ctrliq/ascender-ee` GitHub repository directly (or a fork of it), the included GitHub Actions workflow can build and push the image to GitHub Container Registry automatically on push.

### Step 6: Add the EE to Ascender

1. In the Ascender UI, navigate to **Administration > Execution Environments > Add**
2. Give the Execution Environment a name, for example "Custom EE with `pynetbox`"
3. Set the **Image** field to the full registry path, such as `your-registry.example.com/custom-ascender-ee:v1`
4. Select a **Pull** policy appropriate for your environment
5. If the registry requires authentication, select the appropriate **Credential**
6. Click **Save**

### Step 7: Assign the EE to your Job Template

Edit the Job Template that needs the custom Python packages and set the **Execution Environment** field to the new EE you created. Any jobs launched from this template now use the custom image, and the additional Python packages are available.

## References & related articles

[Ascender Execution Environments documentation](https://ascender-automation.org/userguide/execution_environments.html)  
[Ansible community Getting Started with Execution Environments](https://docs.ansible.com/projects/ansible/latest/getting_started_ee/index.html)  
[CIQ Ascender EE source repository](https://github.com/ctrliq/ascender-ee)


Building a Custom Execution Environment for Ascender


## Introduction

In Ascender, the "Debug Web Requests" setting (Settings > Troubleshooting) enables Python cProfile profiling on every web request. If the required profile output directory does not exist or is not writable, enabling this setting will immediately cause all web requests to fail with **A server error has occurred.**

## Resolution

Since the Web UI is inaccessible, disable the setting via the API using `curl`:

```bash
curl -k -X PATCH \
  -H "Content-Type: application/json" \
  -u admin:<password> \
  https://<ascender-host>/api/v2/settings/debug/ \
  -d '{"AWX_REQUEST_PROFILE": false}'
```

Replace `admin:<password>` with valid Ascender admin credentials and `<ascender-host>` with the Ascender hostname.

Alternatively, if API access is also unavailable, exec into the task pod to run this:

```bash
kubectl exec -n ascender -it deployment/ascender-app-task -c ascender-app-task -- awx-manage shell -c \
  "from awx.conf.models import Setting; Setting.objects.filter(key='AWX_REQUEST_PROFILE').delete()"
```

The Web UI should recover shortly after disabling the setting.

## Root Cause

When this setting is enabled, the `TimingMiddleware` activates cProfile on every incoming request and writes profile output to `/var/log/tower/profile/`. If that directory does not exist or the application user lacks write permissions, the middleware raises an unhandled exception on every response, resulting in a 500 error for all requests, including the UI itself.

## References & related articles

[Quick Start Guide for Installing Ascender on K3s](/article/ascender/asc-ascender-quick-start)


Enabling "Debug Web Requests" Causes Ascender Web UI to Error


## Introduction

Ascender includes a "Debug Web Requests" setting that turns on per-request profiling. Enabling it on a running instance can take the Web UI offline. This article covers how to recover when that happens.

## Problem

After toggling "Debug Web Requests" to ON, the Web UI stops loading and Ascender is effectively unusable. The setting is normally changed from the Web UI, so with it unavailable there is no obvious way to turn it back off.

## Symptoms

Every Web UI page returns the following error right after the setting is enabled:

```text
A server error has occurred.
```

## Resolution

The REST API keeps working even when the Web UI is down, so you can disable the setting through it. "Debug Web Requests" maps to the `AWX_REQUEST_PROFILE` setting, which lives under `/api/v2/settings/debug/`. Set it back to `false`:

```bash
curl -X PATCH \
    -H "Content-Type: application/json" \
    -u admin \
    https://<ascender-host>/api/v2/settings/debug/ \
    -d '{"AWX_REQUEST_PROFILE": false}'
```

Replace `<ascender-host>` with your Ascender host. Passing `-u admin` without the password makes curl prompt for it, which keeps it out of your shell history and the process list; you can also authenticate with an API token using `-H "Authorization: Bearer <token>"` instead. If Ascender uses a self-signed certificate, add `--cacert <ca.pem>` to verify against your CA, or `-k` to skip verification (less secure, avoid on untrusted networks). Once the PATCH succeeds, the Web UI loads normally again.

## Root Cause

"Debug Web Requests" enables per-request profiling, which writes profiling output to `/var/log/tower/profile/`. When that directory doesn't exist, the profiling step errors on every request. Disabling `AWX_REQUEST_PROFILE` stops the profiling attempt and restores normal operation.

## Notes

**_⚠️ WARNING_** _Avoid enabling request profiling or other debug settings on a production Ascender instance._

The REST API remaining available while the Web UI is down is useful more generally: other settings under `/api/v2/settings/` can be read and patched the same way when the UI is unreachable.

If you do need request profiling, confirm that `/var/log/tower/profile/` exists and is writable by the service before enabling the setting.

## References & related articles

[Ascender Documentation](https://docs.ciq.com/ascender)  
[Ascender Installation Overview](https://docs.ciq.com/ascender/1/installation)  
[Ascender API Reference (Community Documentation)](https://ascender-automation.org/rest_api/index.html)


Ascender Web UI Down After Enabling Debug Web Requests


## Introduction

When installing Ascender on Amazon EKS using the `ascender-install` script, the cluster setup is driven through Ansible. Unlike a local RKE2 or K3s install, the EKS path shells out to the AWS CLI to create and query AWS resources (IAM policies, the load balancer, EBS storage). If the AWS CLI is not present where the installer expects it, the run fails partway through. This article covers that failure and its prerequisites. It applies to the `ctrliq/ascender-install` EKS installation path.

## Problem

When installing on EKS (`k8s_platform` set to `eks`), the installer fails on the task that looks up the load balancer IAM policy:

```text
TASK [k8s_setup : Query for AWSLoadBalancerControllerIAMPolicy policy ARN]
fatal: [ascender_host]: FAILED! => {"changed": false,
"cmd": "aws iam list-policies --query 'Policies[?PolicyName==`AWSLoadBalancerControllerIAMPolicy`].Arn' --output yaml",
"msg": "[Errno 2] No such file or directory: b'aws': b'aws'", "rc": 2}
```

The `[Errno 2] No such file or directory: b'aws'` message means Ansible could not find the `aws` executable when it tried to run it. Either the AWS CLI is not installed, or it is installed somewhere the installer does not look.

## Symptoms

- The install fails at `TASK [k8s_setup : Query for AWSLoadBalancerControllerIAMPolicy policy ARN]` with `[Errno 2] No such file or directory: b'aws'`.
- `aws --version` returns `command not found`, or `aws` is installed but not on the installer's `PATH`.

## Resolution

Install the AWS CLI so it is on the installer's `PATH`, configure credentials, then re-run the install.

First check whether the CLI is present:

```bash
aws --version
```

If that returns `command not found`, install the AWS CLI v2 following the [AWS CLI Linux install instructions](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#cliv2-linux-install). The EKS install also needs the `unzip` package for that installer:

```bash
sudo dnf install unzip -y
```

The installer runs `aws` from `PATH` and prepends `/usr/local/bin` to it, which is where the AWS CLI v2 installer places the binary by default, so a standard install works with no extra step. If `aws --version` works in your shell but the install still cannot find it, the installer is running with a different `PATH` than your interactive shell. Put the binary in `/usr/local/bin`, which the installer always adds to its `PATH`, for example with a symlink:

```bash
sudo ln -sf "$(command -v aws)" /usr/local/bin/aws
```

Configure the CLI with an IAM user that has programmatic access (an access key and secret) and permission to create and manage an EKS cluster:

```bash
aws configure
aws sts get-caller-identity   # confirm the active identity
```

Re-run the installer from the repository root once the CLI is in place:

```bash
./setup.sh
```

## Notes

- **EKS requires more than a local install.** A local RKE2 or K3s install never touches AWS, so it does not need the AWS CLI. The EKS installer uses `aws` and `eksctl` to provision and configure cluster resources, which is why these prerequisites apply only to the EKS path. `eksctl` itself does not need to be installed beforehand; the installer sets it up.
- **The install host must be Enterprise Linux 9 for EKS**, not EL8. Confirm with `cat /etc/os-release`.
- The IAM user needs specific permissions to create the cluster and its policies (EKS, EC2, CloudFormation, IAM, and others). The full list, along with a helper playbook to apply them, is in the [Ascender EKS installation guide](https://docs.ciq.com/ascender/1/installation/install-eks).

## Root Cause

The `k8s_setup` role runs `aws iam list-policies` to find or create the `AWSLoadBalancerControllerIAMPolicy` the cluster needs for external load balancers. Ansible executes `aws` as a command and resolves it through `PATH`. The installer prepends `/usr/local/bin` (the default AWS CLI v2 location) to `PATH`, so if `aws` is installed there, or anywhere else on `PATH`, the task runs. When `aws` is not installed, or lives in a directory that is not on `PATH`, the command module returns `[Errno 2] No such file or directory` and the play fails.

## References & related articles

[Ascender EKS installation guide](https://docs.ciq.com/ascender/1/installation/install-eks)  
[Update the Ascender Execution Environment on Kubernetes](/article/ascender/asc-update-execution-environment)


Ascender EKS Install Fails Because the AWS CLI Is Not Found


## Introduction

On security-hardened hosts, a long-running Ansible task can fail partway through with the target host suddenly becoming unreachable, even though the task itself is working correctly. This is common on systems that apply a hardening profile such as DISA STIG or CIS, where systemd is configured to terminate idle login sessions after a fixed period. This article explains why it happens and how to structure your playbook so that long tasks survive the timeout.

This applies to any Ansible run over SSH against a hardened Enterprise Linux host. It is not specific to any single package, task, or Ansible version. It is especially relevant to Rocky Linux from CIQ (RLC) Pro Hardened, where applying the STIG-aligned hardening controls is an available option that can enable this session timeout.

## Problem

Ansible connects to managed hosts over SSH and, for the duration of a task, waits for the remote module to return. When a task takes a long time (large package transactions, image builds, bulk file copies, database imports), the SSH session sits and waits with no interactive traffic.

On a hardened host, systemd-logind may be configured to close sessions it considers idle. When it does, it tears down the login session that Ansible is running under, the SSH connection drops, and Ansible reports the host as unreachable. The task was progressing normally right up to the moment the session was killed, which makes the failure look intermittent and unrelated to the work being done. On RLC Pro Hardened this setting comes from the STIG-aligned hardening controls when they are applied, so the behavior appears once that hardening is in place rather than on a default install.

The relevant setting lives in `/etc/systemd/logind.conf` (or a drop-in under `/etc/systemd/logind.conf.d/`):

```ini
[Login]
StopIdleSessionSec=900
```

A value of `900` terminates sessions after 15 minutes of perceived inactivity. Any single task that runs longer than this window is at risk.

### Notes and Warnings

**_ℹ️ NOTE_** _This is enforced by the operating system, not by SSH. Client-side SSH keepalives (`ServerAliveInterval`, `ClientAliveInterval`) and connection reuse settings (`ControlMaster`, `ControlPersist`) do not prevent it, because systemd-logind is closing the session regardless of SSH-level traffic._

## Symptoms

A task fails with an unreachable error while other hosts in the same play, running the identical task, complete without issue:

```text
TASK [packages : Apply pending updates] ****************************************
changed: [node001]
changed: [node002]
fatal: [node003]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Shared connection to 10.0.0.30 closed.", "unreachable": true}
```

The failure tends to correlate with the hosts that have the most work to do (the largest transaction, the biggest copy) rather than with any particular host or package. Re-running the play often succeeds, because the second run has less work left and finishes inside the timeout window.

You can confirm the timeout is in effect on a target host:

```bash
grep -ri StopIdleSessionSec /etc/systemd/logind.conf /etc/systemd/logind.conf.d/ 2>/dev/null
```

## Resolution

The goal is to run the long operation in a way that does not depend on the login session staying open. The most reliable approach is to launch the work in a transient systemd service unit, which runs under `systemd` (PID 1) rather than under your SSH login session, and then poll that unit for completion so the playbook still tracks the result.

Instead of running a long command directly in a task:

```yaml
- name: Apply pending updates
  ansible.builtin.dnf:
    name: "*"
    state: latest
```

Launch it in a named transient service unit, then wait for that unit to finish and check how it ended:

```yaml
- name: Launch pending updates in a detached transient unit
  ansible.builtin.command:
    cmd: systemd-run --unit=ansible-os-update dnf -y upgrade
  changed_when: true

- name: Wait for the update unit to finish
  ansible.builtin.command:
    cmd: systemctl is-active ansible-os-update
  register: update_unit
  until: update_unit.stdout not in ["active", "activating"]
  retries: 60
  delay: 60
  changed_when: false
  failed_when: false

- name: Fail if the update unit did not complete cleanly
  ansible.builtin.fail:
    msg: "The update unit ended in state '{{ update_unit.stdout }}'"
  when: update_unit.stdout == "failed"
```

Running `systemd-run` without `--scope` starts the command as a transient `.service` unit owned by `systemd` (PID 1), so it is no longer part of the SSH login session. If systemd-logind closes the session, the unit keeps running to completion. `systemd-run` returns as soon as the unit has started, so the play does not hold the connection open for the whole job. The wait task polls `systemctl is-active` until the unit is no longer running, and the final task fails the play if the unit ended in the `failed` state. A failed transient unit is retained by systemd until it is reset, so that state is always available to check.

Avoid `systemd-run --scope` for this. A scope unit is executed by `systemd-run` as its own child and inherits the caller's environment, so it stays inside the login session and can be torn down along with it. The default service mode is what actually detaches the work.

This pattern generalizes to any long operation, not just package updates. For example, a bulk copy or extract launches the same way, with its own unit name:

```yaml
- name: Extract large dataset in a detached transient unit
  ansible.builtin.command:
    cmd: systemd-run --unit=ansible-dataset-extract tar -xzf /tmp/dataset.tar.gz -C /srv/data
  changed_when: true
```

You then wait on `ansible-dataset-extract` with the same `systemctl is-active` check shown above.

Because the unit is detached from the session, you do not have to wait on it right away. If a job's runtime is unpredictable, or you want to start it on many hosts and reconcile later, launch the unit, do other work, and poll for completion further down the play:

```yaml
- name: Start long migration in a detached transient unit
  ansible.builtin.command:
    cmd: systemd-run --unit=ansible-migration /opt/scripts/migrate.sh
  changed_when: true

- name: Do other work while the migration runs
  ansible.builtin.debug:
    msg: "Migration started, continuing with other tasks"

- name: Wait for the migration to finish
  ansible.builtin.command:
    cmd: systemctl is-active ansible-migration
  register: migration_unit
  until: migration_unit.stdout not in ["active", "activating"]
  retries: 180
  delay: 60
  changed_when: false
  failed_when: false
```

Because the work lives in a systemd unit rather than an Ansible async job, an SSH session that drops between starting the unit and polling it does not affect the running job. The unit keeps going, and the next poll picks up its state.

### Alternative approaches

There are other ways to address this, with different trade-offs:

Raise or disable the idle timeout on the managed hosts. You can set a larger `StopIdleSessionSec` (or `0` to disable it) via a drop-in file. This is the simplest change, but on a STIG- or CIS-hardened fleet the timeout is usually mandated by a compliance control, so relaxing it may put the host out of compliance. Prefer a drop-in under `/etc/systemd/logind.conf.d/` over editing `logind.conf` directly, since the main file is package-owned and can be overwritten on update.

**_⚠️ WARNING_** _Changing `StopIdleSessionSec` on a hardened host alters a security control. Confirm with whoever owns the hardening baseline before doing this, and apply it through a drop-in file rather than editing the package-owned `logind.conf`._

Split the work into smaller tasks. Breaking one long transaction into several shorter ones can keep each task inside the timeout window. This avoids touching the security configuration, but it is fragile: as the amount of work grows over time, individual tasks can drift back over the limit, and not every operation can be cleanly subdivided.

The transient service unit approach (`systemd-run` without `--scope`) is the recommended option because it survives the timeout without modifying the security baseline and works for any long-running command.

## Root Cause

Ansible holds the SSH login session open while it waits for a module to return. A hardened host running systemd-logind with `StopIdleSessionSec` set treats that waiting session as idle and terminates it once the configured interval elapses. The task is not failing on its own merits; the session it depends on is being closed out from under it. Detaching the work into a transient systemd service unit removes that dependency, so the operation completes regardless of what happens to the login session.

## References & related articles

[systemd-run manual (Rocky Linux 9)](https://rockyman.org/9.7/systemd/man1/systemd-run.1.html)  
[Ansible asynchronous actions and polling](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_async.html)


Ansible Tasks Killed by systemd Idle Session Timeout on Hardened Hosts


## Introduction

SSH pipelining reduces the runtime of Ascender job templates that run many short tasks. Enabling it takes one configuration change, with no additional software on the managed hosts and no execution environment rebuild. In the benchmark below it cut runtime by a third on a local network, and the benefit grows with network latency.

Set `ANSIBLE_PIPELINING` to `true` under Settings, then Jobs, then Extra Environment Variables. Then use verbose job output to confirm pipelining is active.

Ascender runs playbooks through the standard Ansible `ssh` connection plugin, so this applies to every job template that reaches a host over SSH. The same settings work in a plain `ansible-playbook` run outside Ascender.

The overhead is paid per task and per host, so the total saving scales with the size of the inventory.

## Problem

By default, Ansible needs several SSH operations to run one module. It resolves the remote temporary directory path, creates the directory, transfers the module payload with `sftp`, adjusts permissions, executes the module, then removes the directory. For a plain module such as `command` or `stat`, with no privilege escalation or escalation to root, that is six operations at the SSH layer.

The exact count depends on the module, the privilege escalation settings, and the Ansible version. Each operation adds at least one network round trip, a local SSH client process, and a remote channel setup. For modules that finish in milliseconds, that overhead can dominate the runtime of a playbook containing hundreds of small tasks.

With pipelining enabled, Ansible sends the module payload to the remote Python interpreter through standard input. That avoids the temporary directory, the file transfer, and the cleanup, reducing six SSH operations to one.

Pipelining does not reduce the number of SSH _connections_. Ansible already reuses one connection per host by default, through the `ControlMaster` and `ControlPersist` options in its default `ssh_args`. Pipelining reduces the number of _operations_ carried through that connection. The benchmark below measures connection reuse and pipelining separately.

## Resolution

### Enable pipelining

Pipelining is a connection setting and does not require an execution environment rebuild. Enable it at the instance, project, or inventory level.

To enable pipelining across Ascender, add the following job environment variable under Settings, then Jobs, then Extra Environment Variables. It applies to every playbook run on the instance, regardless of job template or execution environment:

```json
{"ANSIBLE_PIPELINING": "true"}
```

For a single project, set it in the project's `ansible.cfg`. Ascender runs playbooks with the project directory as the working directory, so this file is honored:

```ini
[ssh_connection]
pipelining = True
```

To enable pipelining for selected hosts, set the connection variable in the Variables field of an Ascender inventory, group, or host. This supports a staged rollout and comparison against hosts that do not have it enabled. You can also set the variable in play `vars`:

```yaml
ansible_pipelining: true
```

To confirm pipelining is active, set the job template's Verbosity field to 3 (Debug) and relaunch. A pipelined module task shows no `PUT` or `sftp` lines in the job output.

### Expected gain

Measured against one managed host on a local network, 45 module executions per pass, changing only the two connection settings:

| Connection reuse | Pipelining | Runtime |
| --- | --- | --- |
| Off | Off | 91.0 s |
| On (Ansible default) | Off | 31.3 s |
| Off | On | 31.2 s |
| On (Ansible default) | On | 21.0 s |

Against a stock configuration, which already has connection reuse enabled, pipelining cut runtime by 33 percent. The gain increases with network latency to the managed hosts.

## Notes

### Not every task is pipelined

Pipelining applies only to standard Python modules. Action plugins that transfer files, including `copy`, `template`, `unarchive`, `assemble`, and `script`, continue to create a remote temporary directory and transfer their content whether pipelining is enabled or not. A playbook dominated by `copy` and `template` tasks will see little improvement. Long-running tasks such as large package transactions or image builds also gain little, because SSH overhead is a small part of their runtime.

Tasks using `async` also bypass pipelining, as do modules that are not standard Python modules. Pipelining is disabled outright when `ANSIBLE_KEEP_REMOTE_FILES` is set, and the `su` become plugin does not support it at all.

### The `requiretty` caveat

Historically, `requiretty` prevented pipelining from working with privilege escalation. Without pipelining, Ansible passes `-tt` to allocate a pseudo-terminal, which satisfies the `requiretty` setting in `sudoers`. With pipelining, the module payload occupies standard input, so Ansible cannot use `-tt`. Under `Defaults requiretty`, `sudo` then fails with an error indicating that a terminal is required. That compatibility problem is why pipelining ships disabled.

Current Enterprise Linux releases do not enable `requiretty` by default. It is off by default in upstream sudo and is not set in the shipped `sudoers` file on Rocky Linux 8, 9, or 10. RHEL removed it from the default configuration in version 7.3. Hosts upgraded in place from older releases may retain the setting, so check before enabling pipelining across the fleet:

```bash
sudo grep -r requiretty /etc/sudoers /etc/sudoers.d/ 2>/dev/null
```

`requiretty` and `use_pty` are separate settings. Hardening profiles such as CIS and STIG mandate `Defaults use_pty`, and sudo 1.9.14 and later enable it by default. When sudo is not attached to a terminal, as with pipelining, `use_pty` has no effect and does not interfere with pipelining.

### Keep the default `ControlPersist` setting

A dropped multiplexed connection can cause a broken pipe error, and Ansible may not retry the operation successfully. Keep `ControlPersist` at its default of 60 seconds.

## References & related articles

[Ansible ssh connection plugin documentation](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/ssh_connection.html)  
[Ansible configuration reference](https://docs.ansible.com/ansible/latest/reference_appendices/config.html)  
[Ascender job templates, including the Verbosity field](https://ascender-automation.org/userguide/job_templates.html)  
[Ascender inventories, groups, and host variables](https://ascender-automation.org/userguide/inventories.html)  
[Ansible Tasks Killed by systemd Idle Session Timeout on Hardened Hosts](/article/ascender/asc-long-task-session-timeout)


Speeding Up Ascender Jobs with SSH Pipelining


## Introduction

This document outlines how to update the SSL certificate for an existing Ascender installation. This guide will provide you with two options to update the certificate. It assumes that you have a valid and working Ascender installation. The recommended method for installation is the [ascender-install](https://github.com/ctrliq/ascender-install) script from the CIQ repository. The first certificate update method uses the Ascender installation scripts.

Obtaining SSL certificates for your environment is outside the scope of this article, and you will need to have them issued before proceeding. The Common Name (CN) field in the certificate must match the URL you have chosen for Ascender. Your certificate and private key must be in PEM format, with the certificate stored in a `.crt` (or `.cer`) file and the private key stored in a separate `.key` file.

## Problem

Depending on your certificate provider, you will need to update the SSL certificate for your Ascender install every 90 days to 1 year if you are using a domain validated certificate.

## Resolution

When updating your certificate, you can choose to rerun the setup script or manually update the secret within Kubernetes.

### Rerun the setup script

When you install Ascender using the [ascender-install](https://github.com/ctrliq/ascender-install) scripts provided by CIQ, a `custom.config.yml` file is generated that contains the variables used to perform your initial install. Modify the following variables within this file:

* **kube_install**: Configure this as `false`, as you already have a Kubernetes cluster on which Ascender is running.
* **download_kubeconfig**: Modify this to `false`, as you already have a valid `KUBECONFIG` file to authenticate to your existing Ascender instance.
* **tls_crt_path**: Change this, if necessary, to the location of your certificate.
* **tls_key_path**: Edit this if required, to the place where you stored your private key.

Now, rerun the `setup.sh` script and the new certificate will be installed.

### Modify the Kubernetes secret

To manually deploy the certificate within Kubernetes, you need to create a new secret manifest. Specifically, we will modify the `ascender-tls-secret` file. First, we need to `base64` encode our certificate and private key:

```bash
ASCENDER_CERT=$(base64 -w0 /path/to/crt)
ASCENDER_KEY=$(base64 -w0 /path/to/key)
```

Next, we will create a new manifest using this information:

```bash
cat <<EOF > ascender_tls.yml
apiVersion: v1
kind: Secret
metadata:
  name: ascender-tls-secret
  namespace: ascender
type: kubernetes.io/tls
data:
  tls.crt: ${ASCENDER_CERT}
  tls.key: ${ASCENDER_KEY}
EOF
```

You can apply this by running the following:

```bash
kubectl apply -f ascender_tls.yml
```

After a few seconds, your new certificate will be applied. You can verify your new certificate, by running the following:

```bash
echo | openssl s_client -connect your-ascender-hostname:443 2>/dev/null | openssl x509 -noout -subject -ext subjectAltName -dates
```

## References & related articles

[CIQ Ascender](https://ciq.com/products/ascender-pro)  
[Ascender documentation](https://docs.ciq.com/ascender/)  
[Ascender Install Scripts and Documentation](https://github.com/ctrliq/ascender-install)  
[Ascender Feature Overview](https://ciq.com/blog/manage-enterprise-infrastructure-with-ascender-by-ciq-what-is-ascender/)  
[Ascender Installation Video](https://www.youtube.com/watch?v=j15pFNfs8dw)


Update SSL Certificates for Ascender


## Introduction

Ascender runs project syncs and job templates inside an execution environment (EE) container. When a new EE image is published, for example to pick up a security fix, a Kubernetes-based Ascender deployment does not load it automatically. The running pods keep using the image they already pulled. This article shows how to force Ascender to pull the updated EE. It applies to Ascender Pro deployed on Kubernetes.

## Problem

You force Ascender onto a new EE for one of two reasons.

The most common is to pull a patched EE, usually to pick up CVE remediation in a newer image. The other is recovery: if the current EE can no longer complete project updates or job templates, for example after an upstream change affects one of the dependencies it relies on, pulling the updated image restores the environment.

## Resolution

Confirm the image pull policy, then restart the task deployment so Kubernetes re-pulls the EE.

First, check the pull policy so a restart actually fetches the new image. Replace `ascender` with your namespace and `ascender-app` with your Ascender resource name if they differ:

```bash
kubectl -n ascender get awx ascender-app -o=jsonpath='{.spec.image_pull_policy}'
```

This should return `Always`. With any other policy, Kubernetes reuses the cached image and a restart alone will not pick up the update (see Notes).

Then restart the task deployment to re-pull the EE and load the patched container:

```bash
kubectl -n ascender rollout restart deployment/ascender-app-task
```

***⚠️ WARNING*** *This restarts the Ascender task pods, which interrupts any jobs currently running. Perform it during a maintenance window or when the environment is idle.*

Once the new pods are running, re-run a project sync to confirm it now uses the new EE. If a broken EE was blocking syncs or templates, they should run normally afterward.

## Notes

- If the pull policy is not `Always`, Kubernetes will not re-pull an image it already has cached, so the restart will not pick up the patched EE. Set `image_pull_policy` to `Always` on the Ascender resource (this is configured at install time through the [Ascender install script](https://github.com/ctrliq/ascender-install)) and allow the operator to reconcile before restarting the deployment.
- This procedure refreshes the EE that ships with Ascender. If you maintain your own EE image, see [Building a Custom Execution Environment for Ascender](/article/ascender/asc-custom-execution-environment).

## Root Cause

Kubernetes pulls a container image when a pod first starts and then caches it. A running Ascender task pod keeps using the EE image it started with, even after a new image is published under the same tag. With `image_pull_policy` set to `Always`, restarting the deployment forces Kubernetes to re-pull the current image, so the replacement pods come up on the patched EE.

## References & related articles

[Building a Custom Execution Environment for Ascender](/article/ascender/asc-custom-execution-environment)  
[Update SSL Certificates for Ascender](/article/ascender/asc-update-ascender-ssl)


Update the Ascender Execution Environment on Kubernetes


## Introduction

[Fuzzball](https://ciq.com/products/fuzzball/) Orchestrate is architected as a collection of microservices deployed in a Kubernetes (K8s)
cluster. To understand what is happening with the Fuzzball deployment, you must be able to inspect
the status of the pods supporting these containerized microservices, and gather relevant logs.

## Checking pod status

Any Fuzzball troubleshooting starts with confirming the presence and inspecting the status of
containers/microservices within their respective pods.

You can view and inspect the status of pods hosting containerized microservices using the standard
K8s control plane tool, `kubectl`. After executing the following `kubectl` command, a table
is displayed giving information about all of the pods within the K8s cluster.

```text
# kubectl get pods -A
NAMESPACE            NAME                                                       READY   STATUS      RESTARTS      AGE
cert-manager         cert-manager-56f459fcb7-j7wmt                              1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-56f459fcb7-m8gq2                              1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-56f459fcb7-qj9zd                              1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-cainjector-78f497d9d8-5fc5r                   1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-webhook-5bf5cccd59-qg5qs                      1/1     Running     1 (35m ago)   19d
cert-manager         trust-manager-778f7b488-gdmrh                              1/1     Running     1 (35m ago)   19d
database             database-debug-0                                           1/1     Running     1 (35m ago)   19d
database             database-postgresql-0                                      1/1     Running     1 (35m ago)   19d
fuzzball-system      fuzzball-operator-controller-manager-55cc86985f-c7tzn      2/2     Running     2 (35m ago)   19d
fuzzball             audit-archive-28836720-4wcf5                               0/1     Completed   0             2d6h
fuzzball             audit-archive-28838160-86zjk                               0/1     Completed   0             30h
fuzzball             audit-archive-28839600-tbhj2                               0/1     Completed   0             6h25m
fuzzball             fuzzball-account-7749678b7c-m272g                          1/1     Running     1 (35m ago)   19d
fuzzball             fuzzball-account-external-5f5cd6f56-86k4z                  1/1     Running     1 (35m ago)   19d
fuzzball             fuzzball-account-worker-7f69c98747-gbtvc                   1/1     Running     4 (33m ago)   19d
fuzzball             fuzzball-admin-0                                           1/1     Running     1 (35m ago)   19d
[...snip]
```

If you want to view only pods running within the `fuzzball` namespace, you can change the command
like so:

```text
# kubectl get pods -n fuzzball 
NAME                                             READY   STATUS      RESTARTS       AGE
audit-archive-28836720-4wcf5                     0/1     Completed   0              2d8h
audit-archive-28838160-86zjk                     0/1     Completed   0              32h
audit-archive-28839600-tbhj2                     0/1     Completed   0              8h
fuzzball-account-77bc99544-n2dgb                 1/1     Running     0              119s
fuzzball-account-external-7977f8c5dc-hmfhf       1/1     Running     0              119s
fuzzball-account-worker-6b646c8b76-jq2sf         1/1     Running     0              119s
fuzzball-admin-0                                 1/1     Running     1 (173m ago)   19d
fuzzball-audit-74bb656b6b-lltx4                  1/1     Running     0              119s
fuzzball-auth-spicedb-57bb79cf9b-k2zh7           1/1     Running     0              119s
fuzzball-dns-569f675947-cf59z                    1/1     Running     0              119s
fuzzball-kube-68fc8d8b8d-f7gkg                   1/1     Running     0              119s
fuzzball-log-845fc88dd9-bvr4c                    1/1     Running     0              119s
fuzzball-openapi-686488649-7fj2s                 1/1     Running     0              119s
fuzzball-organization-7f7f5f6878-qs55p           1/1     Running     0              119s
fuzzball-organization-external-cc78884d5-z56j5   1/1     Running     0              118s
fuzzball-organization-worker-55599cd8d8-htknd    1/1     Running     0              118s
fuzzball-permission-external-55d965c957-n5fsv    1/1     Running     0              118s
fuzzball-schedule-0                              1/1     Running     0              170m
[...snip]
```

Note the values in the `READY` column indicating `1/1`, (or potentially `2/2`, etc.) and the
`STATUS` column indicating `Running`. Any pods that do not report a `Running` or `Completed` status,
or those that report fewer instances than expected should be investigated further by inspecting
logs.

## Retrieving and Inspecting logs

While provisioning Fuzzball within a K8s cluster (using `kubectl apply -f fuzzball.yaml`), you can
watch the log output with the following command:

```text
# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1
```

You can stream logs from an individual pod using the following syntax. Note that the namespace must
be specified since Fuzzball pods are not deployed in the "default" namespace. For example, to view
the logs streaming from the `fuzzball-schedule-0` pod, you could use the following command. (Omit
the `--follow --tail=0` options/arguments if you want to view the log at a single instant without
streaming it.)

```text
# kubectl logs -n fuzzball fuzzball-admin-0 --follow --tail=0
```

The `kubectl logs` command outputs verbose logs in json format. These can be filtered and
parsed into human-readable output using `jq` like so:

```text
# kubectl logs -n fuzzball fuzzball-admin-0 | \
    jq -r 'select(.level != "debug") | [.ts, .msg, .Error] | @tsv'
```

The `kubectl` utility does not provide a straightforward command for capturing all Fuzzball logs in
a single stream. However, the logs for a collected Fuzzball “app” can be inspected using a label
selector like so. (This is the strategy used above to stream logs during the provisioning process.)

```text
# kubectl logs --prefix --namespace=fuzzball --all-containers \
    --max-log-requests=99 --tail=0 --follow \
    --selector='app.kubernetes.io/name=fuzzball-schedule'
```

The list of available labels can be viewed using `kubectl get pods -n fuzzball --show-labels`.

Finally, it may be useful to gather a full log dump (for example, to send to CIQ support). This can
be accomplished by requesting logs from all app labels. The app labels may change with different
versions of Fuzzball and can be inspected as described above.  Here is an example:

```text
# kubectl logs --prefix --all-containers --max-log-requests=99 \
    --namespace=fuzzball \
    --selector='app.kubernetes.io/name in (fuzzball-account, fuzzball-admin, fuzzball-audit, fuzzball-auth, fuzzball-dns, fuzzball-kube, fuzzball-log, fuzzball-openapi, fuzzball-organization, fuzzball-permission, fuzzball-schedule, fuzzball-secret, fuzzball-storage, fuzzball-sync, fuzzball-ui, fuzzball-user, fuzzball-workflow)'
```

## Restarting Fuzzball

You can invoke the command `kubectl rollout restart` to restart the Fuzzball deployment. For
example:

```text
# kubectl rollout restart deployment -n fuzzball
```

Other namespaces that may be pertinent for restart include

- `database` - provides the PostgreSQL database that backs Fuzzball Orchestrate state
- `ingress` - maps HTTP(S) URLs to relevant Fuzzball services, including the Fuzzball UI, Fuzzball API, and Keycloak
- `kyverno` - provides a policy engine used by Fuzzball Orchestrate
- `workflow` - provides the temporal service that is used by Fuzzball Orchestrate
- `identity` - provides a Keycloak instance for identity and access management

## Uninstalling Fuzzball

If a deployment failed or you have a need to uninstall a deployed Fuzzball installation, you may
need to tear down the installation and start fresh. The following command will delete an existing
Fuzzball deployment, allowing you to start over if needed.

```text
# kubectl delete FuzzballOrchestrate fuzzball-orchestrate
```

You can watch the deprovisioning process as it occurs with the following command:

```text
# kubectl logs -n fuzzball-system -l app.kubernetes.io/name=fuzzball-operator -f --tail=-1
```

You may also want to remove the `substrate` configuration directory that is shared out to compute
nodes via NFS.  For instance:

```text
# rm -rf /srv/fuzzball/shared/substrate
```

This directory will be recreated if you re-provision your Fuzzball instance.

## References & related articles

[Fuzzball Documentation](https://docs.ciq.com/fuzzball/)


Basic Troubleshooting Procedures


## Introduction

The [Fuzzball](https://ciq.com/products/fuzzball/) Web GUI allows you to view logs from your running or completed job.

## Problem

Under some circumstances, a user may click on the "Logs" button within the job section of the
workflow dashboard and receive the message "No log output" even though the job completed
successfully and logs were expected. The problem may be that the user does not have the appropriate
permissions to view the logs, or that the browser cannot access the API endpoint because Fuzzball
was installed using self-signed certificates.

## Symptoms

Here is an example showing the problem. The user has clicked on the "Logs" button for a workflow
that was expected to produce logs and is seeing the message "No log output".

![No log output in GUI](/images/articles/no-logs.png)

## Resolution

### Ensure you have permission to view logs

First, ensure that you have the appropriate permissions to view logs in the workflow.  If you are
viewing a job in a workflow submitted by another member of your shared account, you might need to
ask them for permission to view the logs.  They can add you as a job owner using the following CLI
command:

```text
fuzzball workflow add-owner <workflow UUID> <user UUID>
```

The workflow UUID and user UUID can be obtained with the `fuzzball workflow list` and `fuzzball
account list-members` commands respectively.  After you have been added as an owner you should be
able to view logs from jobs within the workflow.

### Self-signed certificates require an exception

If you have the correct permission to view job logs, you should check to make sure that your browser
can access them from the API.  When self-signed certificates are used, the Fuzzball GUI asynchronous
access to the Fuzzball Orchestrate API can be blocked by your web browser's security policy, causing
an erroneous indication that no logs are available.

To correct this issue, you can direct your browser to the Fuzzball API endpoint.  The exact URI will
depend on the way that Fuzzball was installed, but it may be accessible by changing the string `ui`
in the Fuzzball URL to `api`.  For instance the following refers to the URL for the Fuzzball web GUI
and the API on an example installation respectively.

- <https://ui.stable.fuzzball.ciq.dev/>
- <https://api.stable.fuzzball.ciq.dev/>

In the case of a self-signed certificate, your browser may complain that the URL is unsafe and may
prompt you to create an exception.  After doing so, a browser refresh should allow you to view the
logs in the web GUI.


Troubleshooting Missing Job Logs in Fuzzball


## Introduction

If no certificates are supplied by the Administrator during the configuration and installation
procedure, the Fuzzball Operator will create self-signed certificates for hosting Fuzzball
Orchestrate.

## Problem

You may run into issues using the Fuzzball CLI if the cluster (context) you are accessing is using
a self-signed certificate.

## Symptoms

If you receive an error like the following when attempting to use the Fuzzball CLI it suggests that
your environment is not properly configured to recognize the self-signed TLS certificates in use.

```text
tls: failed to verify certificate: x509: certificate signed by unknown authority
```

## Resolution

An administrator can use the following commands to export the relevant certificates from the
Fuzzball K8s installation:

```text
# mkdir certs

# kubectl get secret -n cert-manager root-ca-cert -o "jsonpath={.data['ca\.crt']}" | base64 --decode >certs/ca.crt

# kubectl get secret -n cert-manager root-ca-cert -o "jsonpath={.data['tls\.crt']}" | base64 --decode >certs/tls.crt
```

These certificates can be distributed to the systems where the Fuzzball CLI is installed and then
used by adding an environment variable like so:

```text
export SSL_CERT_DIR=/path/to/certs
```

Alternately, CLI users can set an environment variable, `FUZZBALL_INSECURE=true` or use the
`--insecure` flag when invoking the Fuzzball command.

## Notes

The solutions selected here are appropriate for development clusters, testing, and debugging, but it
is ultimately more secure for administrators to use certificates issued by third parties. The
[Fuzzball documentation](https://docs.ciq.com/fuzzball/) contains a section describing how to do this.


Using the CLI with Self-Signed TLS Certificates


## Introduction

When importing existing volumes into [Fuzzball](https://ciq.com/products/fuzzball/) with `fuzzball volume provisioner scan`, the scan can fail immediately with a Postgres `invalid input syntax for type uuid` error. This article explains why the error happens and how to resolve it. It applies to Fuzzball 4.0.x and 4.1.0.

## Problem

Running a provisioner scan to import pre-existing volumes fails on the first entry and no volumes are imported:

```text
fuzzball volume provisioner scan <provisioner-name>
```

```text
rpc error: code = Unknown desc = importing volume "<volume-name>": ERROR: invalid input syntax for type uuid: "" SQLSTATE 22P02
```

The scan aborts on the first entry it processes, so none of the intended volumes are imported.

## Symptoms

- `fuzzball volume provisioner scan` returns `invalid input syntax for type uuid: ""` with `SQLSTATE 22P02`.
- The scan aborts on the first volume it processes, so the specific volume named in the error varies by environment.
- `fuzzball volume list` shows none of the expected volumes after the scan.

## Resolution

The error means the CLI session has no active group (account) selected, so there is no account UUID to attach the imported volumes to. Select a group, then re-run the scan.

First, check your current context and confirm whether a group is active:

```text
fuzzball context list
```

List the groups available to you:

```text
fuzzball group list
```

Set the group you want the imported volumes to belong to:

```text
fuzzball group use <group>
```

Re-run the provisioner scan:

```text
fuzzball volume provisioner scan <provisioner-name>
```

The scan now completes and the volumes import successfully. Confirm with:

```text
fuzzball volume list
```

## Root Cause

Volumes in Fuzzball belong to an account (group), and the account is stored in a `NOT NULL` UUID column in the backing Postgres database. When no group is selected in the CLI session, the scan attempts to write an empty string (`""`) into that UUID column. Postgres rejects the empty string as an invalid UUID and returns `SQLSTATE 22P02`, which surfaces as the `rpc error` above. Selecting a group gives the scan a valid account UUID to associate the volumes with, and the import proceeds.

## References & related articles

[Troubleshooting Missing Job Logs in Fuzzball](/article/fuzzball/fb-no-job-logs)


Fuzzball Volume Provisioner Scan Fails With Invalid UUID Error


## Introduction

Bash keeps a per-user command history in `~/.bash_history`. When `HISTTIMEFORMAT`
is set, bash also records and displays a timestamp for each command. A common
point of confusion arises when a history file contains a mix of entries: some
written before `HISTTIMEFORMAT` was ever set (no stored timestamp) and some
written after (with a stored timestamp). Under those conditions, old commands can
appear to have run "today," show up with unexpectedly high line numbers, and have
no matching login session. This looks alarming, but it is a display artifact, not
evidence that a command was just executed.

This article explains the behavior, how to confirm it, and how to stop the
confusing output. The behavior is general to bash on any Linux distribution,
including [Rocky Linux](https://docs.ciq.com/rlc/).

## Problem

An administrator reviewing root's command history sees a command they did not
expect, for example a `grub2-mkconfig` invocation, listed with the current date
and a high line number. There is no corresponding login in the login records, and
the same entry appears to advance to a new line number every time someone logs in.
The natural conclusion is that the command was run without authorization. In
reality, the entry is an old command with no stored timestamp being displayed
against the current time.

## Symptoms

The following observations tend to appear together:

- A command shows the current date in `history` output, but the file it would have
  modified has a much older modification time. For example, `history` reports
  `grub2-mkconfig` running today, while `ls -l` on the target `grub.cfg` shows a
  modification time weeks earlier.
- The line number for a given entry keeps changing (advancing) on each login.
- No matching session appears in `last` for the time the command supposedly ran.
- No history-clearing or history-rewriting logic is present in `~/.bashrc` or
  `~/.bash_profile`.
- The history file contains two regions: an older block of plain command lines
  with no timestamps, followed by a block where each command is preceded by a
  comment line holding a Unix epoch value.

Inspecting the raw file shows the two regions directly:

```text
# older region, written before HISTTIMEFORMAT was set (no timestamps)
cp /etc/default/grub /etc/default/grub-backup
vi /etc/default/grub
grub2-mkconfig -o /boot/grub2/grub.cfg

# newer region, written after HISTTIMEFORMAT was set (epoch timestamps)
#1690380695
vi /etc/sysctl.conf
#1690380736
```

## Resolution

First, confirm the file has a mix of timestamped and un-timestamped entries by
looking at the raw contents. The un-timestamped commands are the ones that will
display incorrectly:

```bash
grep -n -B5 -A5 grub2-mkconfig ~/.bash_history
```

If you want to stop the confusing output, clear the history for the affected user.
Back the file up first so you retain a record, then truncate it. This is the same
pattern many sites already run against non-root accounts on a schedule:

```bash
cp -p ~/.bash_history ~/.bash_history.bak.$(date +%F)
: > ~/.bash_history
history -c
```

`history -c` clears the in-memory history for the current shell so the old entries
do not get written back out on logout. New commands recorded from this point will
carry a proper `#<epoch>` timestamp. Close any other shell sessions the user has
open before truncating; each open session holds its own in-memory copy of the old
history and can write it back to the file on exit, repopulating what you cleared.

To record timestamps consistently going forward, set `HISTTIMEFORMAT` in a shell
startup file. Adding it to `~/.bashrc` covers that user's interactive shells; to
apply it for all users and shell types, add it under `/etc/profile.d/` instead:

```bash
grep -q HISTTIMEFORMAT ~/.bashrc || echo 'export HISTTIMEFORMAT="%F %T "' >> ~/.bashrc
```

Note that this only affects how bash displays timestamps and whether it records
them for new commands. It cannot retroactively assign a real time to entries that
were written without one.

If your goal is to audit who ran a privileged command and when, do not rely on
shell history. Use the audit daemon, which records the actual execution with a
real timestamp and the login UID:

```bash
auditctl -w /usr/sbin/grub2-mkconfig -p x -k grub_command
ausearch -k grub_command --start today
```

The `auditctl` rule above applies to the running system only. To make it persist
across reboots, add the same rule to a file under `/etc/audit/rules.d/` (for
example `/etc/audit/rules.d/grub.rules`) and reload with `augenrules --load`.

## Root cause

Bash does not store a timestamp with a command unless `HISTTIMEFORMAT` is set at
the time the command is written to history. When it is set, bash writes a separate
line to `~/.bash_history` consisting of `#` followed by the Unix epoch, immediately
before the command it applies to.

If `HISTTIMEFORMAT` was enabled at some point after the history file already
contained commands, those earlier commands have no stored `#<epoch>` line. When
`history` renders them, it has no real time to display, so it falls back to the
current time. That is why a legitimately old command appears to have run today.

Two more bash behaviors explain the rest of the symptoms. Bash does not sort
history by timestamp; it preserves the order in which entries were loaded and
recorded. When history from multiple shells is merged (for example via
`history -n` or `history -r` on login), older commands can end up interleaved and
carry higher line numbers than newer ones. And if the file is never cleared, it
keeps accumulating across every session, so the same old entry keeps getting
re-read and re-numbered on each login.

The takeaway is that shell history is a convenience feature, not an audit trail.
It reflects what commands are stored in a file and how bash chooses to display
them, not a reliable record of what ran and when.

## Notes

Clearing history removes a troubleshooting aid, so keep the backup copy if you may
need to review past commands. For any real security or compliance requirement,
configure the audit daemon (`auditd`) with rules for the paths and commands you
care about. It records the actual system call, the real timestamp, and the login UID,
none of which shell history guarantees.

## References & related articles

[Bash Reference Manual: Bash History Facilities](https://www.gnu.org/software/bash/manual/bash.html#Bash-History-Facilities)  
[Bash Reference Manual: Bash Variables (HISTTIMEFORMAT)](https://www.gnu.org/software/bash/manual/bash.html#Bash-Variables)


Why Old Commands Show Incorrect Dates in Bash History After Enabling HISTTIMEFORMAT


## Introduction

CIQ provides [Long Term Support](https://ciq.com/services/long-term-support/) for Rocky Linux minor releases that have reached End of Life. This article explains how CIQ maintains and updates packages in the LTS repository.

## Standard repositories vs. LTS repository

### Standard repositories

When a Rocky Linux minor release reaches End of Life, for example Rocky 9.2, CIQ freezes all standard repositories in their final state and makes them available in Portal:

- Rocky Linux 9.2 BaseOS - `rocky-9.2-baseos.x86_64`
- Rocky Linux 9.2 AppStream - `rocky-9.2-appstream.x86_64`
- Rocky Linux 9.2 extras - `rocky-9.2-extras.x86_64`

These repositories and others aren't updated after End of Life. They contain the original package versions from that release to satisfy package installation and dependencies.

### LTS repository

CIQ maintains a separate LTS repository where patched package updates are published, for example:

- Rocky Linux 9.2 CIQ Long Term Support - `rlc-9.2-lts.x86_64`

When CIQ patches packages for CVEs or critical bugs, CIQ publishes the updated packages to the LTS repository.

## How to Use LTS Repositories

To find patched packages, use the LTS repository instead of standard repositories when using an LTS product.

### Locating patched packages

In the CIQ customer portal, navigate to the LTS repository for your Rocky Linux version:

```text
https://portal.ciq.com/products/88eb2f09-3146-49c3-b0ef-d2ea9705258e/repository/rlc-9.2-lts.x86_64
```

Browse to the appropriate package directory, for example `/Packages/s/` for packages starting with 's'.

### Installing from LTS

If you configure the LTS repository on your system, patched packages automatically update during regular updates because they have higher version numbers than the frozen standard repositories:

```bash
dnf update
```

## Notes

- LTS repositories are actively maintained throughout the support lifecycle.
- Standard repositories remain frozen for reference and reproducibility.
- Security-patched versions referenced in CIQ CSAF advisories are always in the LTS repository.
- Patched packages in LTS have higher version numbers and automatically upgrade over frozen packages during `dnf update`.

## References & related articles

[CIQ Customer Portal](https://portal.ciq.com)
[Rocky Linux Documentation](https://docs.rockylinux.org)
[Rocky Linux Long Term Support](https://ciq.com/services/long-term-support/)


Understanding CIQ Long Term Support Repositories


## Introduction

The need for secure and efficient file transfer solutions has become paramount. Whether it is sharing sensitive data, collaborating on projects, or automating workflows, a reliable Secure File Transfer Protocol (SFTP) server can streamline these processes, while ensuring data integrity and confidentiality.

The CIQ Secure File Exchange (SFE) service is an SFTP, FTP/S, HTTP/S, and WebDAV server. It provides a flexible foundation for managing file transfers while offering an efficient and scalable solution for storing and retrieving data. The service supports private virtual folders per user, which can also be shared with other users as well.

## How to access the service

To have an SFE account generated, please create a ticket via the [CIQ Support Portal](https://portal.ciq.com). A member of support will then generate your account and securely send you the login credentials via the ticket.

Login to the [CIQ SFE Client Portal](https://sftp.ciq.com/web/client/login) using your provided credentials.

You will then be prompted to change your password. Please do so and store the new password in a secure location.

## How to upload files

Files can be uploaded in two ways: via your web browser or the CLI.

### Web browser method

Log in with your credentials at the [CIQ SFE Client Portal](https://sftp.ciq.com/web/client/login).

Navigate to the `My Files` section and press the `Upload files` button.

![The CIQ SFE Client's My Files section and highlighting the "Upload" button that can be clicked near the top left-hand corner of the screen, just below the word "show".](/images/articles/ciq_sfe_file_upload_1.webp)

Select your files and then hit `Submit`:

![Highlighting the area where the user can upload their files and then clicking the "Submit" button to start the upload.](/images/articles/ciq_sfe_file_upload_2.webp)

The file will be uploaded to your virtual folder. Please inform the Support Team via your support ticket and they can retrieve the files and start their investigation.

**If you would like to further customize how you share your files, please follow the steps below:**

Under `My Files`, click the checkboxes next to the files you uploaded and then click the `Share` button:

![The "My Files" section, with the files that the user wants to share having their checkboxes ticked on the left-hand side of the screen, in the same column as the "Upload" button.](/images/articles/ciq_sfe_file_upload_4.webp)

![The "My Files", highlighting the "Share" button to click that is on the left of the refresh button of the top of the screen.](/images/articles/ciq_sfe_file_upload_3.webp)

At the next screen, add a `Name` for your share, set a `Password` and `Expiration` date if you desire and then select `Submit`:

![The "Add a new share" section of the CIQ SFE Client, showing a "Name" can be added for the share near the top of the page, the "Scope" can be changed from "Read" to "Write" to "Read and Write" under the "Name" section, a Password that can be added around the middle of the page, as well as setting an Expiration date just below the Password entry field. Finally the "Submit" button is highlighted in green for the user to click and the share can be created.](/images/articles/ciq_sfe_file_upload_5.webp)

Then in the `Shares` screen, select the share that you created and click the `Link` button at the top of the page:

![The test share that was created in the previous step and which has been selected for sharing by the user.](/images/articles/ciq_sfe_file_upload_7.webp)

![The "Shares" section of the CIQ SFE Client, showing the "Link" button in the top left-hand corner under the word "entries", where links to the share can be generated.](/images/articles/ciq_sfe_file_upload_6.webp)

Finally, copy the link type you would like to share and include the link in the relevant ticket. The Support Team can then easily access your files:

![Once the "Link" button is clicked, three links are generated: 1) The user can download the share as a zip file. 2) The user can access a directory to view all of the files. 3) If only a single file is shared, the user can download the file uncompressed.](/images/articles/ciq_sfe_file_upload_8.webp)

### CLI method

Run the following command to access the CIQ SFE service via the CLI. Your username is usually your email address:

```bash
sftp -P 2022 <YOUR_USERNAME>@sftp.ciq.com
```

Enter your password when prompted.

Create a directory where you would like to store your files:

```bash
sftp> mkdir <YOUR_DIRECTORY>
```

Upload your files using the `put` command:

```bash
put ./text.txt ./test/
```

Please check the example below for uploading a file called `test.txt` to a directory in the CIQ SFE service called `test`:

```bash
sftp> put ./test.txt test/
Uploading ./test.txt to /test/test.txt
test.txt                                                                                                                                      100%   13     0.1KB/s   00:00
sftp>
```

Once uploaded, please check that the files were uploaded successfully. An example is shown below:

```bash
sftp> ls -l test/
-rwxr-xr-x    1 991      991            13 Jan 20 02:32 test.txt
```

Please inform the Support Team via your ticket that the files have been uploaded successfully.


**_ℹ️ NOTE_** _For all methods listed above, **please provide a sha256sum in the ticket for each file that is in the multiple gigabytes**. This allows the Support Team to easily verify if an upload has been successful or not and can continue their investigation without further interruptions to the customer._

If you run into any problems whilst uploading files to the CIQ SFE service, please don't hesitate to make the Support Team aware and they will assist you with the upload process.


How to Upload Files to the CIQ Secure File Exchange (SFE) Service


## Introduction

This article is a great reference for customers looking to learn more about how CIQ evaluates and remediates security vulnerabilities. 

## FAQ 

**What CVE’s are addressed by CIQ Bridge and CIQ LTS?**  
For both [CIQ Bridge](https://ciq.com/products/ciq-bridge/) and [RLC Pro LTS](https://ciq.com/services/long-term-support/) subscriptions, CIQ focuses on resolving CVEs with CVSS scores of 7 and above. CIQ also considers customer impact and environmental factors. CIQ evaluates CVE’s, taking into account the OS, the build environment, and other Linux vendors' scores to determine the applicability of the CVE to our products. CIQ prioritizes CVE’s based on CVSS score, package exposure, popularity, confidence in the potential fixes (vs. risk of them introducing bugs), and customer requests.

**What does the CVSS Score mean?**  
CIQ evaluates scores from [NIST CVSS 3.x Scoring System](https://nvd.nist.gov/vuln-metrics/cvss), [MITRE](https://www.cve.org/) and other OS Vendors to evaluate and rank vulnerabilities in a standardized and repeatable way. These scores are used as inputs for CIQ to determine the appropriate impact of the CVE to our products.

**Are there SLA’s for CVE remediation?**  
CIQ does not provide SLA’s for any CVE remediation. They are prioritized based on the criteria discussed above.

**Where can I see the status of a given CVE?**  
All fixes can be found in the CSAF files provided by the public [CIQ Advisories Repository](https://github.com/ctrliq/advisories). CVE fixes are also published in the [CIQ Portal](https://portal.ciq.com/cve?sort=updated&direction=desc). For any other status updates, please reach out to your Customer Success Manager or open a Support ticket.

**How do CSAF files relate to the output of my scanning tool?**
Scanners use CSAF files and data from OS vendors to determine if a CVE is fixed based on the version of the RPM. Scanner vendors are responsible for processing Rocky Linux and CIQ CSAF files and advisories to determine if a fix has been released for a given CVE. If your scanner is not ingesting the CSAF files from Rocky/CIQ LTS, it will produce false positives.

**How do CVE’s relate to Security Advisories?**  
Security advisories can contain one or more CVE’s. CIQ evaluates CVEs individually and not Security Advisories. CVE’s will be fixed based on the criteria mentioned above.


CIQ CVE Remediation - Frequently Asked Questions


## Introduction

When you need help, the CIQ Portal is your starting point for creating a support request. Submitting a ticket there ensures your request reaches our Support Team, who will review, respond, and work to resolve the issue as quickly as possible in line with the agreed [Service Level Agreements (SLAs)](https://ciq.com/legal/sla). Please note that all support requests are managed in accordance with [CIQ's terms and conditions](https://ciq.com/legal/terms-conditions/).

## How to create a ticket

### Ticket creation via CIQ Portal

This is the primary way to generate a ticket and make us aware of your issue. Log into the CIQ Portal at [https://portal.ciq.com](https://portal.ciq.com)

**_ℹ️ NOTE:_** _If you are unable to log in to the Portal for any reason, please contact your CSM and they will assist you with access._

Once logged in, click the `Support` button on the left-hand menu. You will see a grid of your existing tickets. To create a new ticket, click the green `+ Create Support Request` button.

Provide the following information:

* `Subject` - a brief description of the problem.

* `Product` - the product you are having trouble with. The available options include Rocky Linux, Warewulf Pro, Warewulf, RLC, and more.

* `Version` - select the version of the product you are experiencing the issue with from the drop-down menu.

* `Priority` - this ranges from `Urgent` to `Low`. Please consult CIQ's [terms and conditions](https://ciq.com/legal/terms-conditions/) before selecting the `Priority`, to ensure that CIQ can provide you with the appropriate level of support.

* `Add collaborators` - optionally choose who to include from your organization on the ticket.

* `Description` - provide as much detail as possible surrounding the issue.
    **_ℹ️ NOTE:_** _If you can [generate an sosreport](https://kb.ciq.com/article/rocky-linux/rl-generate-an-sosreport) or attach additional outputs to help describe the problem to the ticket, that would greatly speed up the resolution process and avoid the Support Team having to ask for additional information._

* `Attachments` - files of up to `50MB` in size can be attached directly to the ticket. If your file type is not supported, please compress it into a `.zip`, `.tar`, or `.gz` file. For files larger than `50MB`, please visit [sftp.ciq.com](https://sftp.ciq.com) for upload. Please ask the Support Team to create a Secure File Exchange account for you.

Once all of the above information is filled out, click the `Submit` button.

You will then be taken to your ticket. All communication with the CIQ Support Team is done here until successful resolution of the issue.

To view all of your tickets, click the `Support` button on the left-hand menu in the CIQ Portal. Under the `Requested` tab, you can see all of the Support Tickets you have created. You can also see all of the tickets from your team members under the `Organization` tab, as well as tickets you have been CC'd in.

### Ticket creation via email

You can also generate tickets by sending an email directly to [support@ciq.com](mailto:support@ciq.com)

The `Subject Line` contains the same information as the `Subject` field when creating a Support Ticket.

The body of the email is the same as the `Description` field listed above in the support ticket creation section.

**_ℹ️ NOTE:_** _Any tickets created by this method are automatically placed as `Normal` (`Severity 3`) `Priority` tickets. If you have an `Urgent` issue, please let the CIQ Support Team know via the ticket and they will raise the `Priority` as needed._

You can also respond to replies from the CIQ Support Team via email as well.

## References & related articles

[Generating sosreports on Rocky Linux](https://kb.ciq.com/article/rocky-linux/rl-generate-an-sosreport)
[How to Upload Files to the CIQ Secure File Exchange (SFE) Service](https://kb.ciq.com/article/general/ciq-secure-file-exchange)
[What Data is Collected in an sosreport on Rocky Linux](https://kb.ciq.com/article/rocky-linux/rl-what-information-an-sosreport-collects)


How to Create a Support Ticket


## Introduction

This article demonstrates how to safely add additional swap space to a Rocky Linux system while it's running and actively using existing swap. This procedure is non-disruptive and doesn't require a system reboot or affect running processes.

## Problem

A system administrator needs to increase available swap space on a server that's currently using its existing swap partition. The system has 2 GB of swap space configured as an LVM partition and requires an additional 10 GB of swap capacity without interrupting any active processes or services.

## Resolution

### Create a Swap File

Instead of modifying existing partition sizes, create a new swap file on the root filesystem. This approach avoids the complexity of resizing LVM volumes and can be done while the system is running.

#### Create the swap file using fallocate

```bash
sudo fallocate -l 10G /swapfile_10gb
```

#### Set appropriate permissions

```bash
sudo chmod 600 /swapfile_10gb
```

#### Format the file as swap

```bash
sudo mkswap /swapfile_10gb
```

#### Enable the new swap file

```bash
sudo swapon /swapfile_10gb
```

This command activates the swap file immediately without affecting your existing swap partition or any running processes.

#### Verify both swap spaces are active

```bash
sudo swapon --show
```

Expected output should show:

- Your original swap partition (e.g., `/dev/mapper/rl-swap` - 2.0GB)
- Your new swap file (`/swapfile_10gb` - 10GB)
- Total: 12 GB of swap space

You can also verify with:

```bash
free -h
```

### Make the Change Persistent

To ensure the system activates the swap file automatically after reboots, add an entry to `/etc/fstab`:

```bash
echo '/swapfile_10gb none swap defaults 0 0' | sudo tee -a /etc/fstab
```

#### Reload systemd configuration

```bash
sudo systemctl daemon-reload
```

## Notes

### Swap Files vs Swap Partitions

Both swap files and swap partitions provide equivalent performance on modern filesystems. Swap files offer several advantages:

- No need to modify partition tables or LVM volumes
- You can create, resize, or remove them without downtime
- Multiple swap files can coexist with swap partitions
- Easier to manage on systems with complex storage configurations

### Filesystem Compatibility

Rocky Linux typically uses XFS for the root filesystem, and both XFS and ext4 fully support non-sparse swap files created with `fallocate`. Modern kernels treat a properly allocated swap file the same as a swap partition for reliability and performance. Using `fallocate` offers advantages over older `dd if=/dev/zero` examples because it's faster and avoids writing unnecessary zeroes.

Note: Advanced or copy-on-write filesystems, for example Btrfs, can require extra steps such as setting the "no copy-on-write" attribute before creating a swap file. This isn't a concern for default Rocky Linux XFS or ext4 deployments and is outside the scope of this article.

### Swap Priority

By default, the kernel uses swap spaces with equal priority. If you need to prioritize one swap space over another, you can specify the `pri` option in `/etc/fstab`, for example:

```bash
/swapfile_10gb none swap defaults,pri=10 0 0
```

The kernel uses higher priority values first. Priority values range from -2 (lowest, the default) to 32767 (highest). By default, if no priority is specified, the kernel assigns a priority of -2.

## References & related articles

[Rocky Linux Documentation Filesystem - swapon swapoff](https://docs.rockylinux.org/labs/systems_administration_II/lab6-the_file_system/?h=swap#swapon-and-swapoff)
[How to Limit Tmpfs for Taking Swap Space](https://kb.ciq.com/article/warewulf/ww-tmpfs-swap-memory)


Adding Swap Space to a Live Rocky Linux System


## Introduction

At CIQ, customers will often create tickets due to a kernel crash they experienced. As part of the investigation process, the Support Team asks for a `vmcore` dump to be uploaded to CIQ's Secure File Exchange (SFE) service for analysis.

For more information on how to upload a `vmcore` dump or other files to the SFE service, please consult the [How to Upload Files to the CIQ Secure File Exchange (SFE) Service](https://kb.ciq.com/article/general/ciq-secure-file-exchange) KB article.

This article will detail some of the steps that can be taken with the Crash Utility for `vmcore` dump analysis.

For further reading, the author highly recommends the following resources:

* [Crash Utility White Paper by David Anderson](https://crash-utility.github.io/crash_whitepaper.html)

* [Linux Kernel Crash Book - Everything you need to know by Igor Ljubuncic](https://www.dedoimedo.com/computers/www.dedoimedo.com-crash-book.pdf)

## Example issue

A customer created a ticket with [CIQ Support](https://portal.ciq.com), after their CentOS 7.9 node using CIQ Bridge had a kernel panic and rebooted. What follows will be how to setup the Crash Utility and the steps taken to debug the issue.

### Crash Utility environment setup

* Set up a Rocky Linux 9.x installation on a node size of your choice.

* Update all packages:

```bash
sudo dnf update -y
```

* Install the `kexec-tools` and `crash` packages:

```bash
sudo dnf install -y kexec-tools crash
```

* From the node that went into a kernel panic state, find the kernel version:

```bash
uname -r
```

* With the kernel version identified, search and download the equivalent version of the `kernel-debuginfo-common` package. In the CentOS 7.9 example listed above, the kernel version was `3.10.0-1160.119.1.el7.x86_64`, thus the required package was `kernel-debuginfo-common-x86_64-3.10.0-1160.119.1.el7.x86_64.rpm`.

* Similarly, the same version of the `kernel-debuginfo` package is also needed. Again referencing the CentOS 7.9 example, downloading the `kernel-debuginfo-3.10.0-1160.119.1.el7.x86_64.rpm` package is required.

* **Note for Rocky Linux users** - to find the appropriate `kernel-debuginfo-common` and `kernel-debuginfo` packages, go to [The Rocky Linux Vault](https://dl.rockylinux.org/vault/rocky/). Using Rocky Linux 9.5 with the x86_64 architecture as an example, the `kernel-debuginfo-common` package is found [in the devel repository](https://dl.rockylinux.org/vault/rocky/9.5/devel/x86_64/os/Packages/k/) and `kernel-debuginfo` [is in the devel repository under debug](https://dl.rockylinux.org/vault/rocky/9.5/devel/x86_64/debug/tree/Packages/k/).

* Install both `kernel-debuginfo-common` and `kernel-debuginfo` packages using the `localinstall` command. The order the packages are installed in also matters:

```bash
sudo dnf install -y <path_to_kernel-debuginfo-common_rpm>
sudo dnf install -y <path_to_kernel-debuginfo_rpm>
```

* **Note for CentOS users** - the `vmcore` dump will be generated as a `vmcore.flat` file. This is not natively readable by the Crash Utility and thus has to be converted into a `vmcore` dump file. Perform the following command to convert the file:

```bash
/usr/sbin/makedumpfile -R ./vmcore < ./vmcore.flat
```

* Finally, start the Crash Utility by setting the `vmlinux` version as an argument under your `/usr/lib/debug/lib/modules/<kernel_version>` directory and pointing to the `vmcore` dump file:

```bash
crash /usr/lib/debug/lib/modules/<kernel_version>/vmlinux <path_to_vmcore_dump>
```

### Example crash dump analysis

* Starting the Crash Utility will bring up an overview of the `vmcore` dump, including the date the dump was taken, node name, memory availability, kernel version, and the kernel panic reason.

* An example kernel panic that CIQ often sees from customers running Kubernetes is `blocked tasks`:

```bash
"Kernel panic - not syncing: hung_task: blocked tasks"
```

* From the initial overview, the amount of memory available was plentiful during the kernel panic. However, as you will see later on in the article, having plentiful memory does not rule out a memory-related issue:

```bash
MEMORY: 1011.7 GB
```

* For deeper analysis on memory usage and availability, the `kmem -i` command can be run and produces an output similar to the below example:

```bash
crash> kmem -i
                 PAGES        TOTAL      PERCENTAGE
    TOTAL MEM  261010406     995.7 GB         ----
         FREE  228447706     871.5 GB   87% of TOTAL MEM
         USED  32562700     124.2 GB   12% of TOTAL MEM
       SHARED  1626752       6.2 GB    0% of TOTAL MEM
      BUFFERS     3277      12.8 MB    0% of TOTAL MEM
       CACHED  20211378      77.1 GB    7% of TOTAL MEM
         SLAB   765701       2.9 GB    0% of TOTAL MEM

   TOTAL HUGE        0            0         ----
    HUGE FREE        0            0    0% of TOTAL HUGE

   TOTAL SWAP  1048575         4 GB         ----
    SWAP USED        0            0    0% of TOTAL SWAP
    SWAP FREE  1048575         4 GB  100% of TOTAL SWAP

 COMMIT LIMIT  131553778     501.8 GB         ----
    COMMITTED  26620006     101.5 GB   20% of TOTAL LIMIT
```

* Running the `sys -i` command shows more details about the hardware, as well as BIOS information, and product tags. Sensitive information such as serial numbers have been removed in the following example:

```bash
crash> sys -i
        DMI_BIOS_VENDOR: HPE
       DMI_BIOS_VERSION: A43
          DMI_BIOS_DATE: 12/03/2021
         DMI_SYS_VENDOR: HPE
       DMI_PRODUCT_NAME: ProLiant DL325 Gen10 Plus
    DMI_PRODUCT_VERSION:
     DMI_PRODUCT_SERIAL: -
       DMI_PRODUCT_UUID: -
       DMI_BOARD_VENDOR: HPE
         DMI_BOARD_NAME: ProLiant DL325 Gen10 Plus
      DMI_BOARD_VERSION:
       DMI_BOARD_SERIAL: -
    DMI_BOARD_ASSET_TAG: -
     DMI_CHASSIS_VENDOR: HPE
       DMI_CHASSIS_TYPE: 23
    DMI_CHASSIS_VERSION:
     DMI_CHASSIS_SERIAL: -
  DMI_CHASSIS_ASSET_TAG: -
```

* The `bt` command produces a backtrace output for the process that caused the kernel panic. The succeeding example shows process `650` running command `khungtaskd` was responsible for the kernel panic:

```bash
crash> bt
PID: 650      TASK: ffff9b2ffbb59080  CPU: 47   COMMAND: "khungtaskd"
 #0 [ffff9b2ffb663cb0] machine_kexec at ffffffffb2c69854
 #1 [ffff9b2ffb663d10] __crash_kexec at ffffffffb2d29f12
 #2 [ffff9b2ffb663de0] panic at ffffffffb33ab713
 #3 [ffff9b2ffb663e60] watchdog at ffffffffb2d56bfe
 #4 [ffff9b2ffb663ec8] kthread at ffffffffb2ccb621
```

* More information on `khungtaskd` can be found [at this article](https://www.tencentcloud.com/document/product/213/41974), however a summary can be observed beneath:

> The kernel hung task is based on a single kernel thread named as khungtaskd, which monitors processes in the TASK_UNINTERRUPTIBLE status. If a process stuck in D state during the period specified by kernel.hung_task_timeout_secs (defaults to 120 seconds), the stack information of this hung task process will be printed.

* To verify that the kernel is configured to crash in a `khungtaskd` state, the `p` command (to print the value of an expression) with the `sysctl_hung_task_panic` argument can be used:

```bash
crash> p sysctl_hung_task_panic
sysctl_hung_task_panic = $1 = 1
```

* Similarly the timeout can also be checked using the `p sysctl_hung_task_timeout_secs` command:

```bash
crash> p sysctl_hung_task_timeout_secs
sysctl_hung_task_timeout_secs = $2 = 360
```

* To view more information about a particular process, the `ps` command plus the PID number as an argument can be used:

```bash
crash> ps 650
      PID    PPID  CPU       TASK        ST  %MEM      VSZ      RSS  COMM
>     650       2  47  ffff9b2ffbb59080  RU   0.0        0        0  [khungtaskd]
```

* Finding out more information about which tasks were blocked, can be done with the `log` command and a `grep` for the word `INFO`. In the subsequent example, `systemd` is seen as being blocked:

```bash
crash> log | grep INFO
[3970064.996286] INFO: task systemd:1 blocked for more than 360 seconds.
```

* For showing the stack traces of all blocked tasks in a `TASK_UNINTERRUPTIBLE` state, the `foreach UN bt` command is recommended:

```bash
foreach UN bt
```

* Further filtering is also possible on all blocked tasks with this command string. The command picks tasks #2, #3, and #4 from each process, places them into three columns, counts the amount of stack trace messages occurrences and sorts from highest to lowest:

```bash
foreach UN bt | awk '/#[2-4] /{print $3,$5}' | paste - - - | sort | uniq -c | sort -nr
```

* An example is here:

```bash
crash> foreach UN bt | awk '/#[2-4] /{print $3,$5}' | paste - - - | sort | uniq -c | sort -nr
     46 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     proc_cgroup_show ffffffffb2d35146
     33 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     cgroup_lock_live_group ffffffffb2d2ed49
      2 __mutex_lock_killable_slowpath ffffffffb33b6520 mutex_lock_killable ffffffffb33b65fe    iterate_dir ffffffffb2e71d00
      1 schedule_timeout ffffffffb33b5831       wait_for_completion ffffffffb33b805d    wait_rcu_gp ffffffffb2cc828e
      1 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     proc_cgroupstats_show ffffffffb2d2f2be
      1 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     cgroup_release_agent ffffffffb2d2ffe5
      1 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     cgroup_free_fn ffffffffb2d34308
```

* From the above output, a high volume of `mutex_lock_slowpath` messages can be viewed. Researching the error attributes this to either a hardware issue or low resource availability (CPU / Memory / I/O).

* The full list of unique blocked tasks can be found by running `foreach UN bt | grep COMMAND | awk '{print $NF}' | sort -u`:

```bash
crash> foreach UN bt | grep COMMAND | awk '{print $NF}' | sort -u
"cadvisor"
"dockerd"
"filebeat"
"kubelet"
"kworker/113:0"
"kworker/31:1"
"process-exporte"
"runc"
"systemd"
"systemd-journal"
```

* To dig further into a blocked task, use the `set` command with the PID as the argument. The ensuing example sets the focus on the blocked task `systemd`:

```bash
crash> set 1
    PID: 1
COMMAND: "systemd"
   TASK: ffff9a32db050000  [THREAD_INFO: ffff9a32dbb9c000]
    CPU: 42
  STATE: TASK_UNINTERRUPTIBLE
```

* Then run `bt -l` for more verbose backtrace messages. In the above-mentioned `mutex_lock_slowpath` example, the process that was running before the `mutex_lock` was engaged was `proc_cgroup_show`:

```bash
crash> bt -l
#2 [ffff9a32dbb9fda8] __mutex_lock_slowpath at ffffffffb33b6ca7
    /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/include/linux/spinlock.h: 337
#3 [ffff9a32dbb9fe08] mutex_lock at ffffffffb33b602f
    /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/arch/x86/include/asm/current.h: 14
#4 [ffff9a32dbb9fe20] proc_cgroup_show at ffffffffb2d35146
    /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/kernel/cgroup.c: 4706
```

* The line number from the `cgroup.c` file being referenced by `proc_cgroup_show` was `4706`. It is possible to view the relevant source code from the installed `kernel-debuginfo` packages with the `list` or `l` command:

```bash
crash> l /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/kernel/cgroup.c: 4706
4701
4702            retval = 0;
4703
4704            mutex_lock(&cgroup_mutex);
4705
4706            for_each_active_root(root) {
4707                    struct cgroup_subsys *ss;
4708                    struct cgroup *cgrp;
4709                    int count = 0;
4710
```

* You can see from line `4704` that the `mutex_lock` is related to the `Cgroup` system. From the [kernel.org documentation on Cgroups](https://docs.kernel.org/admin-guide/cgroup-v1/cgroups.html):

> There is a global mutex, cgroup_mutex, used by the cgroup system. This should be taken by anything that wants to modify a cgroup. It may also be taken to prevent cgroups from being modified, but more specific locks may be more appropriate in that situation.

* Since `Cgroups` handle the allocation of resources, the mutex lock observed here points towards a lack of resources on the node.

* As an aside, if the `mutex_lock` code pointed to `mutex_lock(&dentry->d_inode->i_mutex);`, that would highlight an inode-related error on the storage medium and further investigation would be required there.

* Finally, after running the `log` command to dump the `system message buffer`, multiple `Hardware Error` messages were observed related to the customer's memory DIMMs:

```bash
crash> log
<output_redacted>
[3968484.124456] {176586}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 20480
[3968484.124460] {176586}[Hardware Error]: It has been corrected by h/w and requires no further action
[3968484.124461] {176586}[Hardware Error]: event severity: corrected
[3968484.124463] {176586}[Hardware Error]:  Error 0, type: corrected
[3968484.124464] {176586}[Hardware Error]:  fru_text: DIMM# Sourced
[3968484.124465] {176586}[Hardware Error]:   section_type: memory error
[3968484.124466] {176586}[Hardware Error]:   error_status: 0x0000000000040400
[3968484.124468] {176586}[Hardware Error]:   physical_address: 0x000000f877d52000
[3968484.124470] {176586}[Hardware Error]:   node: 0 card: 3 module: 1 rank: 2 bank: 6 row: 57311 column: 0
[3968484.124471] {176586}[Hardware Error]:  Error 1, type: corrected
[3968484.124472] {176586}[Hardware Error]:  fru_text: DIMM# Sourced
[3968484.124473] {176586}[Hardware Error]:   section_type: memory error
[3968484.124474] {176586}[Hardware Error]:   error_status: 0x0000000000040400
[3968484.124475] {176586}[Hardware Error]:   physical_address: 0x000000f7f9b63180
[3968484.124477] {176586}[Hardware Error]:   node: 0 card: 3 module: 1 rank: 2 bank: 6 row: 56806 column: 16
[3968484.124501] [Hardware Error]: Corrected error, no action required.
[3968484.125383] [Hardware Error]: CPU:0 (17:31:0) MC1_STATUS[-|CE|-|-|AddrV|-|-|-|-]: 0x940000000000009f
[3968484.126221] [Hardware Error]: Error Addr: 0x000000f877d52000, IPID: 0x0000000000000000
[3968484.126907] [Hardware Error]: Instruction Fetch Unit Extended Error Code: 0
[3968484.127593] [Hardware Error]: Instruction Fetch Unit Error: microtag probe port parity error.
[3968484.128280] [Hardware Error]: cache level: L3/GEN, tx: RESV
```

## Example issue resolution

Ultimately it was found that the lack of resources were due to `Hardware Errors` in the customer's memory DIMMs, thus causing a contention for resources resulting in tasks being blocked for over 360 seconds and therefore invoking a kernel panic.

## Conclusion

`vmcore` dump analysis is a vast and complicated process. There are almost a limitless number of ways that the kernel can crash and the root cause may not always be the same from system to system. The author hopes that the example analysis above for blocked tasks can help to demystify some of the research required and make it easier to identify a root cause if you run into a similar situation.


Tips and Tricks for Analyzing a vmcore dump with the Crash Utility


## Introduction

This article covers an issue where `/var/log` is more restrictive than the shipped default and reverts on each boot. One potential cause is the audit daemon.

## Problem

An administrator expects to find `/var/log` at `0755`, which is the shipped default, but finds it at `0700` instead. Applications or log collectors that read files under `/var/log` as a non-root user stop working, because a mode of `0700` prevents any non-root account from traversing the directory. Correcting the mode by hand does not survive the next reboot.

## Symptoms

The directory mode is more restrictive than expected:

```bash
stat -c '%a %U:%G %n' /var/log
```

```text
700 root:root /var/log
```

The boot journal shows the audit daemon starting immediately after `systemd-tmpfiles` has set the directory to `0755`:

```bash
journalctl -b | grep -E 'Create Volatile Files|auditd.*Init complete'
```

```text
Starting Create Volatile Files and Directories...
Finished Create Volatile Files and Directories.
auditd[1234]: Init complete, auditd 3.1.5 listening for events (startup state enable)
```

## Resolution

Check where the audit daemon writes its log:

```bash
grep -E '^log_file|^log_group' /etc/audit/auditd.conf
```

A system exhibiting this issue has `log_file` pointing directly into `/var/log` rather than at the default of `/var/log/audit/audit.log`:

```text
log_file = /var/log/auditd.log
log_group = root
```

The audit daemon restricts the directory that holds its log file, so a log file placed directly in `/var/log` causes the daemon to restrict `/var/log` itself. With `log_group` set to `root`, that mode is `0700`.

Edit `/etc/audit/auditd.conf` and restore the default location:

```text
log_file = /var/log/audit/audit.log
```

**_⚠️ WARNING_** _Restarting the audit daemon interrupts audit logging briefly, and the log path changes. Schedule a maintenance window, and update any log shipping or monitoring that reads the old path first._

Restart the daemon. `auditd.service` sets `RefuseManualStop=yes`, so `systemctl restart auditd` is refused and the service wrapper is needed instead:

```bash
service auditd restart
```

The daemon no longer touches `/var/log`, but it does not raise the mode back either, so restore it once by hand. A reboot does the same by way of `systemd-tmpfiles`:

```bash
chmod 0755 /var/log
```

Confirm the result:

```bash
stat -c '%a %U:%G %n' /var/log /var/log/audit
```

```text
755 root:root /var/log
700 root:root /var/log/audit
```

`/var/log` now holds at `0755` across daemon restarts and reboots, and the daemon restricts `/var/log/audit`, which is where that restriction belongs.

## References & related articles

[auditd.conf(5) manual page](https://rockyman.org/9.8/audit/man5/auditd.conf.5.html)  
[tmpfiles.d(5) manual page](https://rockyman.org/9.8/systemd/man5/tmpfiles.d.5.html)


Permissions on /var/log Revert to 0700 on Every Boot


Selecting [Azure Confidential Computing](https://learn.microsoft.com/en-us/azure/confidential-computing/confidential-vm-faq) (ACC) for your workload gives you exceptional security for sensitive workloads on Rocky Linux from CIQ. However, with that added protection comes a few critical things to keep in mind, especially if you're using `Confidential OS Disk Encryption`.

This guide will help you avoid common issues that can lead to inaccessible VMs and downtime. Prevention is simple once you know what to look out for.

## Why this matters

When using encrypted disks, the encryption key is sealed to specific system values (PCRs) during provisioning. These values depend on things like:

* The UEFI firmware

* The Secure Boot shim

* Kernel certificates and signatures

Any unexpected change to these components, even valid ones, can break the key seal. If that happens, your system will fail to unlock the disk automatically, and you'll be stuck at a boot time passphrase prompt via the serial console.

## What not to change

To keep your system booting normally, avoid making changes to Secure Boot related components unless absolutely necessary:

### Don't change the shim

[Rocky Linux from CIQ](https://ciq.com/products/rocky-linux/) uses a `shim` signed by Microsoft and provided by CIQ. Replacing it or using a non-CIQ shim will break the chain of trust.

### Don’t install a custom kernel

Only use kernels installed by default with your CIQ image. Custom kernels, even from Rocky Linux's upstream (RESF) or [kernel.org](https://www.kernel.org/), will likely break the boot process.

### Avoid untrusted drivers or modules

Some drivers or changes to the initrd can affect the PCRs, even if the system still appears secure.

### Set a fallback passphrase

To avoid being permanently locked out, add a custom passphrase to the TPM:

```bash
cryptsetup luksAddKey /dev/sdX --token-type systemd-tpm2
```

Replace `/dev/sdX` with your encrypted disk (use `lsblk` to find it). Store the passphrase somewhere safe.

You could test it by forcing a PCR change:

```bash
cd /boot/efi/EFI/rocky
mv shim_certificate.efi.bak shim_certificate.efi
reboot
```

Then use the passphrase in the serial console when prompted. After testing, restore the original `shim`:

```bash
mv shim_certificate.efi shim_certificate.efi.bak
```

## Notes

Consider creating a separate recovery key. This would allow you to unlock the encrypted volume by attaching the VHD to another VM, handy for emergencies.

Always create your fallback passphrase as one of your first steps when deploying a new instance.

Avoid unsupported kernels or unauthorized Secure Boot changes.

Stick with 5th or 6th gen DC- and EC- Azure instance types for full ACC support.

Being proactive helps you stay secure without getting locked out.

## References & related articles

[Azure Confidential Computing](https://learn.microsoft.com/en-us/azure/confidential-computing/confidential-vm-faq)  
[Data & System Recovery Considerations on Azure Confidential Computing (ACC)](https://ciq.com/blog/data-and-system-recovery-considerations-on-azure-confidential-computing-acc/)


Preventing Boot Issues on Azure Confidential Computing with Rocky Linux


## Introduction

This article addresses an issue where a Rocky Linux server fails to boot and is waiting on mount points. The system drops to emergency mode because LVM physical volumes are not detected, even though some logical volumes remain available.

## Problem

After applying system patches, the server fails to boot normally. The OS drops to emergency mode, reporting missing logical volumes (such as `/var` and `/tmp`). LVM commands do not detect any physical volumes, though some logical volumes (like root and swap) may still be available.

## Symptoms

- System fails to boot into multi-user mode and drops to emergency mode.
- Boot process hangs waiting for logical volumes.
- LVM commands (`pvs`, `pvscan`, `lvmdiskscan`) return no results.
- Some logical volumes are available and mountable.
- When booted from rescue media, `lsblk` shows the logical volumes, but LVM commands still show nothing.

## Resolution

### Boot into Rescue Mode

Boot from any Rocky Linux installation ISO and select `Troubleshooting` from the boot menu. 

Choose `Rescue a Rocky Linux system`. 

Once the new environment is loaded, it will ask if you would like to automatically detect an existing installation. Press `1` to continue to boot into a shell.

### Verify LVM State

Run the following commands to check the state of LVM:

```shell
lvs -a -o +devices
pvdisplay
lsblk
```

If `lsblk` shows the logical volumes but LVM commands do not, proceed to the next step.

### Rebuild LVM Device Settings

Back up the current LVM devices file:

```shell
mv /etc/lvm/devices/system.devices /root/system.devices.backup
```

Recreate LVM device settings:

```shell
vgimportdevices -a
```

Reboot the system:

```shell
reboot
```

After rebooting, the system should detect all LVM physical volumes and boot normally.

## Root Cause

This issue is typically caused by a corrupted or outdated `/etc/lvm/devices/system.devices` file, which prevents LVM from detecting physical volumes after system changes such as patching or kernel updates. Rebuilding the device settings allows LVM to rescan and recognize the correct devices.

## Notes

- LVM backup metadata is stored in `/etc/lvm/backup/` and `/etc/lvm/archive/`. Always ensure these backups are preserved before making changes.
- Removing or moving the `system.devices` file is safe as long as a backup is created.

## References

[Rocky Linux LVM Documentation](https://docs.rockylinux.org/ja/books/admin_guide/07-file-systems/)  
[vgimport man page](https://man7.org/linux/man-pages/man8/vgimportdevices.8.html)


LVM Physical Volumes Not Detected in Rocky Linux


## Introduction

This article outlines the steps required to configure a PXE boot environment for Rocky Linux using Lorax to generate boot image and the Rocky Linux DVD image for creating the BaseOS and AppStream repositories.

This guide uses `httpd` for serving the installation tree, `dnsmasq` to provide DHCP and TFTP, and `lorax` to generate EFI bootable installation media.

This setup enables automated network-based Rocky Linux installations over a local network.

## Problem

Users often need to deploy Rocky Linux at scale without manually installing each system.

Manual DVD-based installation is not feasible in large environments or headless setups.

There is a need to set up a PXE boot environment that supports UEFI systems and provides the Rocky Linux installer via network boot.

## Symptoms

Client machines are unable to PXE boot due to missing or misconfigured TFTP, HTTP, or DHCP services.

A boot fails with errors such as "no DHCP response", "file not found via TFTP", or "missing installation source".

UEFI systems may hang or drop to a shell if the GRUB configuration or bootloader path is incorrect.

## Resolution

### Prerequisites

* `root` access or `sudo` privileges.

* A Rocky Linux DVD ISO.

* Two machines, one as the PXE host and one as the PXE client directly connected to each other / on a segmented network and doesn't have DHCP servers running.

* A working Internet connection.

### Initial setup

* Install these required packages:

```bash
sudo dnf install -y lorax httpd tftp-server dnsmasq
```

* Enable the `httpd` and `tftp` services:

```bash
sudo systemctl enable --now httpd
sudo systemctl enable --now tftp
```

* Download the Rocky Linux 10 DVD ISO (Rocky Linux 10 is the example used throughout this article):

```bash
wget https://download.rockylinux.org/pub/rocky/10/isos/x86_64/Rocky-10.1-x86_64-dvd1.iso
```

* Mount the ISO and copy its contents:

```bash
mkdir /mnt/dvd
sudo mount -o loop ~/Rocky-10.1-x86_64-dvd1.iso /mnt/dvd
sudo mkdir -p /var/www/html/rocky10-dvd
sudo cp -av /mnt/dvd/. /var/www/html/rocky10-dvd/
```

### Boot image generation

* Generate the Rocky Linux 10 boot image:

```bash
sudo lorax --product "RockyLinux" --version "10" --release "10" --source "https://download.rockylinux.org/pub/rocky/10/BaseOS/x86_64/os/" --source "https://download.rockylinux.org/pub/rocky/10/AppStream/x86_64/os/" --isfinal --logfile lorax.log --buildarch x86_64 --volid "RL10_LIVENET" /tmp/lorax-out
```

* Copy the PXE boot files:

```bash
sudo mkdir -p /var/lib/tftpboot/rocky10
sudo cp /tmp/lorax-out/images/pxeboot/{vmlinuz,initrd.img} /var/lib/tftpboot/rocky10/
```

* Copy the installer stage 2 files:

```bash
sudo cp -r /tmp/lorax-out/images/ /var/www/html/rocky10/
```

* Mount the EFI boot image:

```bash
mkdir /tmp/efiboot
sudo mount -o loop /tmp/lorax-out/images/efiboot.img /tmp/efiboot
```

* Copy UEFI boot files:

```bash
sudo mkdir -p /var/lib/tftpboot/rocky10/EFI/BOOT
sudo cp -av /tmp/efiboot/EFI/BOOT/* /var/lib/tftpboot/rocky10/EFI/BOOT/
```

### Grub configuration

* Configure GRUB:

```bash
cat << "EOF" | sudo tee /var/lib/tftpboot/rocky10/EFI/BOOT/grub.cfg
menuentry 'Rocky Linux 10 Install (PXE Boot)' {
    linuxefi /rocky10/vmlinuz inst.stage2=http://192.168.1.150/rocky10 inst.repo=http://192.168.1.150/rocky10-dvd
    initrdefi /rocky10/initrd.img
}
EOF
```

* Configure `dnsmasq` for PXE boot:

```bash
cat << "EOF" | sudo tee /etc/dnsmasq.conf
interface=enp2s0
bind-interfaces
domain-needed
bogus-priv
log-dhcp
dhcp-range=192.168.1.2,192.168.1.249,12h
dhcp-boot=rocky10/EFI/BOOT/grubx64.efi
enable-tftp
tftp-root=/var/lib/tftpboot
EOF
```

* Restart services:

```bash
sudo systemctl restart dnsmasq
sudo systemctl restart httpd
```

* Allow services through the firewall:

```bash
sudo firewall-cmd --add-service=tftp --permanent
sudo firewall-cmd --add-service=http --permanent
sudo firewall-cmd --add-service=dns --permanent
sudo firewall-cmd --add-service=dhcp --permanent
sudo firewall-cmd --reload
```

* Disable SELinux:

```bash
sudo setenforce 0
sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/' /etc/selinux/config
```

## Root cause

PXE boot environments require coordination between DHCP, TFTP, and HTTP services.

Incorrect or missing files, misconfigured GRUB boot paths, or firewall/SELinux restrictions can prevent PXE booting from succeeding.

Without properly generated and served installer images and boot files, UEFI systems cannot retrieve necessary components to launch the installer.

## Notes

This guide is written for UEFI PXE clients only.

Legacy BIOS support would require additional configuration changes to use `pxelinux.0` or GRUB BIOS equivalents.

Ensure that the `interface=` line in `dnsmasq.conf` matches your actual network interface name.

Network clients must be on the same subnet as the PXE server.

## References & related articles

[Creating a Custom Rocky Linux ISO](https://docs.rockylinux.org/guides/isos/iso_creation/)


PXE Boot Installation of Rocky Linux Using Lorax, dnsmasq, and TFTP


## Introduction

CentOS 7 became [End of Life on June 30th, 2024](https://blog.centos.org/2023/04/end-dates-are-coming-for-centos-stream-8-and-centos-linux-7/) and is now no longer supported with any security updates of any kind. In order to make sure that your systems are up-to-date, switching to Rocky Linux is an excellent choice for security and peace of mind.

If you need more time before starting the migration over to Rocky Linux, then [CIQ Bridge](https://ciq.com/products/ciq-bridge/) is the recommended solution for you.

## Prerequisites

- A CentOS 7.9 node to perform the migration on.

---

**_⚠️ WARNING_** _Make sure you are on the latest minor version of CentOS 7 (`7.9`), before attempting the migration._

---

---

**_⚠️ WARNING_** _You are not able to directly go from CentOS 7.9 to Rocky Linux 9.x. You must upgrade to Rocky Linux 8.10. Upgrades between major versions of Rocky Linux (8.x to 9.x) using Leapp [are not recommended](https://docs.rockylinux.org/release_notes/9_7/) and it is better to perform a fresh install._

---

---

**_⚠️ WARNING_** _Ensure that all data is safely backed up in at least three locations, before the migration begins._

---

---

**_⚠️ WARNING_** _While Leapp is the tool being championed in this article, the best method for migration that CIQ recommends is to set up a new node with Rocky Linux 9.7, move your data and applications over to that node, and then perform a cut off when the CentOS 7.9 node is no longer required. Even in a basic CentOS 7.9 migration without any specific applications installed, many blockers can occur during the migration process._

---

- Run the below commands as either `root` or a user with `sudo` privileges:

## Updating the CentOS 7.9 repositories to point towards CentOS Vault

- Overwrite the `CentOS-Base` repo file in `/etc/yum.repos.d/CentOS-Base.repo` with the following:

```bash
cat << "EOF" | sudo tee /etc/yum.repos.d/CentOS-Base.repo
[base]
name=CentOS-$releasever - Base
baseurl=http://vault.centos.org/7.9.2009/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[updates]
name=CentOS-$releasever - Updates
baseurl=http://vault.centos.org/7.9.2009/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[extras]
name=CentOS-$releasever - Extras
baseurl=http://vault.centos.org/7.9.2009/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[centosplus]
name=CentOS-$releasever - Plus
baseurl=http://vault.centos.org/7.9.2009/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
EOF
```

- Upgrade all packages:

```bash
yum upgrade -y
```

- Reboot:

```bash
reboot
```

## Leapp installation

- Set up the ELevate repository:

```bash
yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el$(rpm --eval %rhel).noarch.rpm
```

- Install the `leapp` packages:

```bash
yum install -y leapp-upgrade leapp-data-rocky
```

- Run pre-upgrade checks:

```bash
leapp preupgrade
```

- In the event the `preupgrade` process highlights any issues, check the `/var/log/leapp/leapp-report.txt` file for a list of all issues and `/var/log/leapp/leapp-preupgrade.log` for a full log from the `preupgrade` process.

- In `leapp-report.txt`, what you want to look for is `Upgrade has been inhibited due to the following problems`. The items listed underneath that will prevent the upgrade from starting, even if you run the `leapp upgrade` command.

The following section presents examples of blockers and how they can be addressed:

### Example blockers

#### Missing required answers in the answer file

```bash
Risk Factor: high (inhibitor)
Title: Missing required answers in the answer file
Summary: One or more sections in answerfile are missing user choices: remove_pam_pkcs11_module_check.confirm
For more information consult https://red.ht/leapp-dialogs.
Related links:
    - Leapp upgrade fail with error "Inhibitor: Missing required answers in the answer file.": https://access.redhat.com/solutions/7035321
Remediation: [hint] Please register user choices with leapp answer cli command or by manually editing the answerfile.
[command] leapp answer --section remove_pam_pkcs11_module_check.confirm=True
```

##### Solution

- The `remove_pam_pkcs11_module_check` check in the `/var/log/leapp/answerfile` file has to be set to `True`:

```bash
leapp answer --section remove_pam_pkcs11_module_check.confirm=True
```

- Once done, run the `leapp preupgrade` command again and if there are no other `inhibitors` found, the end summary will become either yellow or green, indicating that the upgrade can proceed.

#### Detected custom leapp actors or files

```bash
Risk Factor: high
Title: Detected custom leapp actors or files.
Summary: We have detected installed custom actors or files on the system. These can be provided e.g. by third party vendors, Red Hat consultants, or can be created by users to customize the upgrade (e.g. to migrate custom applications). This is allowed and appreciated. However Red Hat is not responsible for any issues caused by these custom leapp actors. Note that upgrade tooling is under agile development which could require more frequent update of custom actors.
The list of custom leapp actors and files:
    - /usr/share/leapp-repository/repositories/system_upgrade/common/files/rpm-gpg/8/RPM-GPG-KEY-Rocky-8
Related links:
    - Customizing your Red Hat Enterprise Linux in-place upgrade: https://red.ht/customize-rhel-upgrade
Remediation: [hint] In case of any issues connected to custom or third party actors, contact vendor of such actors. Also we suggest to ensure the installed custom leapp actors are up to date, compatible with the installed packages.
```

##### Solution

- This can be safely ignored.

#### GRUB2 core will be automatically updated during the upgrade

```bash
Risk Factor: high
Title: GRUB2 core will be automatically updated during the upgrade
Summary: On legacy (BIOS) systems, GRUB2 core (located in the gap between the MBR and the first partition) cannot be updated during the rpm transaction and Leapp has to initiate the update running "grub2-install" after the transaction. No action is needed before the upgrade. After the upgrade, it is recommended to check the GRUB configuration.
```

##### Solution

- This can also safely be ignored if you are on a UEFI system.

#### Difference in Python versions and support in RHEL 8

```bash
Risk Factor: high
Title: Difference in Python versions and support in RHEL 8
Summary: In RHEL 8, there is no 'python' command. Python 3 (backward incompatible) is the primary Python version and Python 2 is available with limited support and limited set
of packages. If you no longer require Python 2 packages following the upgrade, please remove them. Read more here: https://red.ht/rhel-8-python
Related links:
    - Difference in Python versions and support in RHEL 8: https://red.ht/rhel-8-python
Remediation: [hint] Please run "alternatives --set python /usr/bin/python3" after upgrade
```

##### Solution

- Move your applications over to using Python 3.

- The Python 2 packages have many other packages that depend on them. The recommendation is to not remove these packages, unless absolutely required.

#### Risk factor: low warnings

- Go through all of these, however usually these can be ignored (one of the warnings, is that `SELinux` will be set to `permissive` mode, so make sure to change that back to `enforcing` if needed):

```bash
sed -i 's/SELINUX=permissive/SELINUX=enforcing/' /etc/selinux/config

setenforce 1
```

## Start the upgrade

- Once the above `inhibitor` issues have been sorted, run this command to start the upgrade process:

```bash
leapp upgrade
```

- Once completed, `reboot` the machine:

```bash
reboot
```

- At the `GRUB` menu, select the `ELevate-Upgrade-Initramfs` option.

- When that section of the installation is complete, the system will reboot into Rocky Linux (you will see the `grub` menu populated with the Rocky Linux kernels).

- `SELinux` will then perform a relabel upon first boot.

- Once the relabel is complete, the system will reboot a second time.

- If all is successful, you will be presented with a login prompt and at the top you will see `Rocky Linux 8.10 (Green Obsidian)`.

- Check for any leftover packages from the migration:

```bash
rpm -qa | grep "el7\."
```

- Remove these with the `dnf remove` command.

## References & related articles

[Rocky Linux Forum User's Experience Upgrading from Rocky Linux 8 to Rocky Linux 9](https://forums.rockylinux.org/t/update-rocky-linux-8-to-9/10338)
[Rocky Linux 9.7 Release Documentation](https://docs.rockylinux.org/release_notes/9_7/)


How to Migrate From CentOS 7.9 to Rocky Linux 8.10 with Leapp

How to Migrate From CentOS 8.5 to Rocky Linux 9.5 with Leapp


## Introduction

CIQ provides QCOW2 cloud images for Rocky Linux, including RLC Pro, LTS, and other variants. These images use cloud-init for first-boot provisioning, which means you need to supply configuration data (user credentials, SSH keys, network settings) before the instance is usable.

This article covers how to build a cloud-init data source and boot a Rocky Linux QCOW2 image using libvirt/KVM, and includes pointers for other virtualization platforms.

## Problem

After importing a Rocky Linux QCOW2 cloud image and booting a virtual machine from it, you can't log in. The image has no default password set and relies entirely on cloud-init to configure user accounts, SSH keys, and networking on first boot. Without a properly configured cloud-init data source, the instance boots but is inaccessible.

## Resolution

Cloud-init reads configuration from a data source attached to the VM. The most common method for on-premises virtualization is a NoCloud ISO containing your user-data and meta-data files, attached as a virtual CD-ROM. You can also include a `network-config` file in the ISO for static IP assignment or other network settings (see the [NoCloud data source documentation](https://docs.cloud-init.io/en/latest/reference/datasources/nocloud.html) for the full format). The VM reads this ISO on first boot, applies the configuration, and marks itself as initialized. Some hypervisors (such as Proxmox and Xen Orchestra) handle this automatically through their own cloud-init integration, so you don't always need to build the ISO yourself.

Rocky Linux 9 ships cloud-init 24.4 and Rocky Linux 8 ships cloud-init 23.4. See the [cloud-init documentation](https://docs.cloud-init.io/en/latest/) for the full list of supported modules and data sources.

The sections below walk through building a NoCloud data source manually with libvirt/KVM, followed by pointers for other platforms.

### Create the cloud-init configuration

Create a `user-data` file with your desired first-boot settings. At minimum, you need a user account with either a password or SSH key:

```yaml
#cloud-config
hostname: rocky
users:
  - name: rocky
    lock_passwd: false
    plain_text_passwd: changeme
    ssh_authorized_keys:
      - ssh-rsa AAAA...your-key-here
    sudo: ALL=(ALL) NOPASSWD:ALL
ssh_pwauth: true
```

Setting `ssh_pwauth: true` allows password-based SSH login, which is useful as a fallback if SSH key delivery fails. You can also include additional directives such as package installation, timezone configuration, or custom commands:

```yaml
#cloud-config
hostname: rocky
users:
  - name: rocky
    lock_passwd: false
    ssh_authorized_keys:
      - ssh-rsa AAAA...your-key-here
    sudo: ALL=(ALL) NOPASSWD:ALL
ssh_pwauth: true

timezone: America/New_York

packages:
  - vim
  - tmux

runcmd:
  - systemctl enable --now cockpit.socket
```

Create a minimal `meta-data` file. This identifies the instance to cloud-init:

```yaml
instance-id: rocky-vm-01
local-hostname: rocky
```

### Build the NoCloud ISO

Use `mkisofs` (provided by the `xorriso` package in AppStream) to generate the NoCloud ISO. The volume label must be set to `cidata` so cloud-init recognizes it as a NoCloud data source:

```bash
dnf install xorriso
mkisofs -output seed.iso -volid cidata -joliet -rock user-data meta-data
```

This produces a `seed.iso` file that contains your cloud-init configuration in the NoCloud format.

### Boot the image with virt-install

Create a working copy of the QCOW2 image and optionally resize it, then launch the VM with the cloud image and seed ISO attached:

```bash
cp Rocky-9-GenericCloud.latest.x86_64.qcow2 rocky-vm.qcow2
qemu-img resize rocky-vm.qcow2 20G

virt-install \
  --name rocky-cloud \
  --memory 2048 \
  --vcpus 2 \
  --import \
  --disk rocky-vm.qcow2 \
  --disk seed.iso,device=cdrom \
  --os-variant rocky9 \
  --network bridge=br0
```

Cloud-init runs during the first boot, applies your user-data configuration, and writes a marker to `/var/lib/cloud/instance/boot-finished`. On subsequent boots, cloud-init recognizes the marker and skips re-initialization.

### Alternative: virt-install --cloud-init

On Rocky Linux 9 and later, `virt-install` has a `--cloud-init` flag that generates a temporary NoCloud ISO automatically and removes it after the first boot:

```bash
virt-install \
  --name rocky-cloud \
  --memory 2048 \
  --vcpus 2 \
  --import \
  --disk rocky-vm.qcow2 \
  --cloud-init root-password-generate=on,ssh-key=/home/user/.ssh/id_rsa.pub \
  --os-variant rocky9 \
  --network bridge=br0
```

The generated root password is printed to the console during creation. This flag is not available on Rocky Linux 8, where the manual `mkisofs` method is required.

### Other virtualization platforms

The same cloud-init principles apply on other hypervisors and cloud platforms. Each has its own mechanism for delivering cloud-init data to the guest. Note that vSphere does not natively support QCOW2, so you need to convert the image to VMDK format with `qemu-img convert` before importing.

- [Proxmox VE cloud-init support](https://pve.proxmox.com/wiki/Cloud-Init_Support)
- [XCP-ng / Xen Orchestra cloud features](https://docs.xcp-ng.org/management/cloud/)
- [OpenStack instance user data](https://docs.openstack.org/nova/latest/user/metadata.html)
- [VMware vSphere cloud-init data source](https://docs.cloud-init.io/en/latest/reference/datasources/vmware.html)

## Notes

If you need to re-apply cloud-init configuration after the initial boot, clear the cloud-init state and reboot:

```bash
cloud-init clean
reboot
```

CIQ support covers the Rocky Linux operating system and the cloud-init package as shipped in the distribution. Hypervisor-specific configuration is outside the scope of CIQ support, but CIQ can assist with cloud-init behavior on the guest side.

## References & related articles

[cloud-init Documentation](https://docs.cloud-init.io/en/latest/)  
[cloud-init NoCloud Data Source](https://docs.cloud-init.io/en/latest/reference/datasources/nocloud.html)


Configuring Cloud-Init for Rocky Linux QCOW2 Images


## Introduction

Customers often need a custom ISO for a litany of reasons - perhaps they require a mirror of a repository or to edit kernel cmdline parameters for their own customers.

## Problem

This guide explains how to create a custom Rocky Linux 8.10 installation ISO.

## Symptoms

The customer did not have clear direction or resources on how to build their own ISO.

## Resolution

- This guide includes a high-level overview of the ISO build process, along with the commands needed to generate a custom ISO.
- A guide on how to create a custom Rocky Linux ISO was also pushed upstream to the [Rocky Linux Docs](https://docs.rockylinux.org/10/guides/isos/iso_creation/). Please find the same guide below:

### Prerequisites

- A 64 bit machine running Rocky Linux 9 to build the new ISO image.
- A Rocky Linux 9 DVD ISO image.
- A `kickstart` file to apply to the ISO.
- Read the Lorax [Quickstart](https://weldr.io/lorax/lorax.html#quickstart) and [mkksiso](https://weldr.io/lorax/mkksiso.html) documentation to become familiar with how the `Anaconda` `boot.iso` is created.

### Package installation and setup

- Install the `lorax` package:

```bash
sudo dnf install -y lorax
```

### Building the ISO with a kickstart file

- Run the `mkksiso` command to add a `kickstart` file and then build a new ISO:

```bash
mkksiso --ks <PATH_TO_KICKSTART_FILE> <PATH_TO_ISO_TO_MODIFY> <OUTPUT_PATH_FOR_BUILT_ISO>
```

- Below is an example `kickstart` file `example-ks.cfg`, which sets up a Rocky Linux 9.5 `Server With GUI` environment:

```text
lang en_GB
keyboard --xlayouts='us'
timezone Asia/Tokyo --utc
reboot
cdrom
bootloader --append="rhgb quiet crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
zerombr
clearpart --all --initlabel
autopart
network --bootproto=dhcp
firstboot --disable
selinux --enforcing
firewall --enabled
%packages
@^server-product-environment
%end
```

### Adding a repository with its packages to an ISO image

- Make sure the repository you want to add has the `repodata` directory inside of it. If not, this can be created using the `createrepo_c` command and this can be installed with `sudo dnf install -y createrepo_c`
- Add the repository to your `kickstart` file, using the following syntax:

```text
repo --name=extra-repo --baseurl=file:///run/install/repo/<YOUR_REPOSITORY>/
```

- Add your repository using the `--add` flag with the `mkksiso` tool:

```bash
mkksiso --add <LINK_TO_YOUR_REPOSITORY> --ks <PATH_TO_KICKSTART_FILE> <PATH_TO_ISO_TO_MODIFY> <OUTPUT_PATH_FOR_BUILT_ISO>
```

- The process is detailed further using the `baseos` repository in the example below:
- The `baseos` repository will be locally downloaded along with all of its packages:

```bash
dnf reposync -p ~ --download-metadata --repo=baseos
```

- Then add the repository to the `kickstart` file:

```text
repo --name=extra-repo --baseurl=file:///run/install/repo/baseos/
```

- The `kickstart` file would look like the following:

```text
lang en_GB
keyboard --xlayouts='us'
timezone Asia/Tokyo --utc
reboot
cdrom
bootloader --append="rhgb quiet crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
zerombr
clearpart --all --initlabel
autopart
network --bootproto=dhcp
firstboot --disable
selinux --enforcing
firewall --enabled
repo --name=extra-repo --baseurl=file:///run/install/repo/baseos/
%packages
@^server-product-environment
%end
```

- Then point the `mkksiso` command directly to the repository directory and build the ISO:

```bash
mkksiso --add ~/baseos --ks example-ks.cfg ~/Rocky-9.5-x86_64-dvd.iso ~/Rocky-9.5-x86_64-dvd-new.iso
```

### Conclusion

What has been discussed here are just a few of the options available to tweak and build your own Rocky Linux ISO. For further ways, including modifying the kernel cmdline arguments, the author highly recommends to go through the [mkksiso](https://weldr.io/lorax/mkksiso.html) documentation in more detail.

## References & related articles

[Anaconda Documentation](https://anaconda-installer.readthedocs.io/en/latest/)\
[Automating the Installation with Kickstart - Fedora Wiki](https://docs.fedoraproject.org/en-US/fedora/f36/install-guide/advanced/Kickstart_Installations/#chap-kickstart-installations)\
[Dracut - Fedora Wiki](https://fedoraproject.org/wiki/Dracut)\
[Lorax Documentation](https://weldr.io/lorax/)\
[mkksiso Documentation](https://weldr.io/lorax/mkksiso.html)\
[Mock Repository](https://github.com/rpm-software-management/mock)\
[Peridot Repository](https://github.com/rocky-linux/peridot)\
[Creating a Custom Rocky Linux ISO - Rocky Linux Docs](https://docs.rockylinux.org/10/guides/isos/iso_creation/)


Creating a Custom Rocky Linux ISO


## Introduction

This guide explains how to synchronize repositories locally using `reposync`. It covers syncing both publicly available repositories and private repositories, such as those within CIQ Depot. This guide will provide you with two options for downloading packages depending on the complexity of your needs.

## Problem

You need to create a local copy of a repository for an air-gapped environment, a network with limited or slow internet access, or to provide faster access to a local set of servers.

## Prerequisites

A web server is required to serve the downloaded packages. You can follow our [guide](https://ciq.com/blog/how-to-set-up-a-web-server-in-rocky-linux/) on how to set up a web server. When syncing repositories, you will also need to consider the storage requirements.

For smaller repositories that only feature updates such as our [Long Term Support (LTS)](https://ciq.com/services/long-term-support/) repository, the overall size can be somewhere in the range of 1GB to 15GB or more in size.

The full repository for Rocky Linux or CentOS can potentially exceed 100GB in total size.

In this guide, the web server will have an IP address of `192.168.1.30`.

## Option 1: Sync on an existing server

If you are using a Rocky Linux server or another server using CIQ's [CentOS Bridge service](https://ciq.com/products/ciq-bridge/) that is already configured for the repository you are looking to sync, you can leverage its existing setup to download the repository.

Identify the repository you want to download by running `dnf repolist`. The first column will provide you with the repo id that you will use in your reposync command.

In this example, CIQ is syncing using the `repoid` of *rocky-lts-9.2.x86_64* attached to our *CIQ LTS for Rocky Linux 9.2* repository.

```bash
dnf reposync -p repodata --download-metadata \
  --repoid=rocky-lts-9.2.x86_64 --download-path=/var/www/html/rocky-lts-9.2.x86_64/
```

If you are trying to sync our [CentOS Bridge](https://ciq.com/products/ciq-bridge/) repository (requires an active subscription) from a CentOS 7 server, the command is different:

```bash
reposync -p repodata --download-metadata \
   --repoid=ciq-bridge.x86_64 --download_path=/var/www/html/ciq-bridge/
```

The above commands will store the repository files in the local web server path `/var/www/html/`. You can also store these files on shared storage for centralized access or copy them to another server.

## Option 2: Sync on a separate server

If you want to run this on a dedicated server or need to sync multiple repositories, consider creating dedicated repo files to manage the information.

### Step 1: Create a repository configuration file

For CIQ repositories, the `depot` client will write the repo file for you. On a mirror host that already has the client installed and logged in with `depot login`, start by listing what your organization has access to:

```bash
depot list
depot list rlc-pro-9
```

Then generate the config for the product you want:

```bash
depot enable rlc-pro-9 --print > /root/repos/rlc-pro-9.repo
```

Use `--print`. Without it, the file goes straight into `/etc/yum.repos.d/` and the product is marked enabled, so your mirror host starts installing from the repos you only meant to copy. With `--print`, the config goes to stdout instead. The filename it would have used goes to stderr, so a plain redirect gives you a clean file.

To take part of a product rather than all of it, name the repos with `--repo`:

```bash
depot enable rlc-pro-9 --print --repo=rlc-9-core.x86_64 --repo=rocky-9-appstream.x86_64 > /root/repos/rlc-pro-9.repo
```

The same config is in the [Customer Portal](https://portal.ciq.com) if you would rather not use the CLI. Pick the product, installation method, version, and architecture, then open the **Reposync** tab under Installation Instructions for a `.repo` file and a matching `dnf reposync` command.

Two things to know about the file before you sync it.

Most of the repositories in it are disabled. A generated RLC Pro 8 file holds 37 repositories and enables 6: BaseOS, AppStream, Extras, and the CIQ Core, Pro and Supplemental repos. Devel, PowerTools or CRB, high availability, resilient storage, NFV, Plus and RT are all off, as is a `.source` and `-debug` pair for each. `reposync` mirrors the enabled ones only, which is a sensible baseline. Add any others at sync time with `--enablerepo=rocky-8-powertools.x86_64`.

The file also holds your organization's credentials. `chmod 600` it and keep it out of git.

For our example, we will use the publicly available Rocky Linux 9 BaseOS repository. We will create a new repository file named `rocky-baseos-9.repo` in our root directory. Feel free to save this in a more permanent location.

```ini
[rocky-baseos-9]
name=Rocky Linux 9 - BaseOS
baseurl=http://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/
gpgcheck=1
enabled=1
countme=1
metadata_expire=6h
gpgkey=https://dl.rockylinux.org/pub/rocky/RPM-GPG-KEY-Rocky-9
```

The repo id is specified at the top of the file between the `[]` characters. In this case, `[rocky-baseos-9]`.

Repeat the above step to create separate files for each repository you plan to sync.

Alternatively, you can add multiple repositories to a single file, ensuring each has a unique repo id.

### Step 2: Run `reposync`

Once the configuration file is created, you can run the `reposync` command as before. Only now you will specify the repo file like in the below example:

```bash
dnf -c ./rocky-baseos-9.repo reposync -v -p repodata \
--download-metadata --repoid=rocky-baseos-9 \
--download-path=/var/www/html/rocky-baseos-9/
```

Repeat this step for each file and/or repository you intend to sync ensuring that you change the file, repo id, and folder as needed.

### Step 3: Verify and configure clients

After all downloads are complete, confirm the repository is accessible via your web browser.

In the below example, our repository is located at `http://192.168.1.30/rocky-baseos-9/rocky-baseos-9/`.

Once you’ve verified that the packages are being served correctly, configure Rocky Linux to use the repository by updating the appropriate file in `/etc/yum.repos.d/`.

As illustrated below, since we are serving the BaseOS repository, we will modify `/etc/yum.repos.d/rocky.repo`.

```ini
[baseos]
name=Rocky Linux $releasever - BaseOS
#mirrorlist=https://mirrors.rockylinux.org/mirrorlist?arch=$basearch&repo=BaseOS-$releasever$rltype
baseurl=http://192.168.1.30/rocky-baseos-9/rocky-baseos-9/
gpgcheck=1
enabled=1
countme=1
metadata_expire=6h
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Rocky-9
```

## Additional Information

### Mirroring more than one major version from a single host

If you are building a patching baseline for 8, 9, and 10, one mirror server can hold all three. `reposync` only cares about the `baseurl` in the repo file, so a Rocky 9 host has no trouble pulling 8 and 10 content. Generate a repo file per major version:

```text
/root/repos/rlc-pro-8.repo
/root/repos/rlc-pro-9.repo
/root/repos/rlc-pro-10.repo
```

Generating them with the CLI has one catch. `depot enable` checks a product's repositories against the OS ID, OS version, and architecture of the host you run it on, and if none of them match you get a 404 rather than a config file:

```console
# depot enable rlc-pro-8
ERROR: failed to enable product: could not obtain dnf repos: failed to get dnf configuration:
bad status code from server: 404: Not Found: no compatible dnf repositories found,
skipped 63 incompatible repos
```

That is the version check failing. The mirror host's own version (9, here) goes into the request, so every repo in the 8 product gets skipped. Override it with `--osversion`:

```bash
depot enable rlc-pro-8  --osid=rocky --osversion=8  --print > /root/repos/rlc-pro-8.repo
depot enable rlc-pro-9  --osid=rocky --osversion=9  --print > /root/repos/rlc-pro-9.repo
depot enable rlc-pro-10 --osid=rocky --osversion=10 --print > /root/repos/rlc-pro-10.repo
```

Architecture works the same way. Add `--arch` when the content is not for the architecture the mirror host runs on:

```bash
depot enable rlc-pro-8  --osid=rocky --osversion=8  --arch=aarch64 --print > /root/repos/rlc-pro-8-aarch64.repo
depot enable rlc-pro-9  --osid=rocky --osversion=9  --arch=aarch64 --print > /root/repos/rlc-pro-9-aarch64.repo
depot enable rlc-pro-10 --osid=rocky --osversion=10 --arch=aarch64 --print > /root/repos/rlc-pro-10-aarch64.repo
```

From there one `reposync` per version does the work:

```bash
#!/bin/bash
set -uo pipefail
rc=0

for ver in 8 9 10; do
  conf=/root/repos/rlc-pro-${ver}.repo
  dnf -c "${conf}" --setopt=reposdir= reposync --download-metadata \
    --download-path="/var/www/html/rocky/${ver}/x86_64/" \
    || { echo "sync failed for ${ver}" >&2; rc=1; }
done

exit "${rc}"
```

`--setopt=reposdir=` is the part not to drop. `dnf -c` replaces the main config but leaves `reposdir` pointing at `/etc/yum.repos.d/`, so without it the mirror host's own repos join the sync and you get Rocky 9 packages in your 8 tree.

Leave `--repoid` off. `reposync` already limits itself to the repos the file enables, and `--repoid` is a shortcut for `--disablerepo="*" --enablerepo=<repoid>`, which brings back whatever you name regardless of the `enabled` setting. `--enablerepo` is the flag for adding one.

`reposync` writes a subdirectory per repo id, so the served path lands at `/var/www/html/rocky/10/x86_64/rlc-10-core.x86_64/`. Keep the versions in separate trees and clients using `$releasever` resolve correctly.

Rocky 10 ships dnf5, where `--download-path` is `--destdir`. Check `dnf reposync --help` on a 10 host before reusing the command above.

Confirm your subscription covers all three versions first. If it does not, you get an authentication failure rather than the 404 above, which looks similar and is a different problem.

### Avoid `--newest-only` with `--download-metadata`

`--newest-only` is the obvious way to keep a mirror small, and the RHEL 7 era scripts most people are migrating from used it. Paired with `--download-metadata` it produces a repository that does not work.

The metadata comes from upstream unchanged, so it still lists the older packages `--newest-only` skipped. Clients that resolve to one of those get a 404 for a package the metadata promised. The man page says as much: "When the option is used with `--newest-only`, only latest packages will be downloaded, but metadata will still contain older packages."

Either sync everything and leave the metadata alone, or use `--newest-only` without `--download-metadata` and build the metadata yourself with `createrepo_c` afterwards. For a patching baseline, sync everything. The disk is cheaper than the upkeep.

### Migrating from a `reposync` and `createrepo` script

RHEL 7 era scripts usually ran `createrepo` after `reposync`, because `reposync` did not fetch the upstream metadata:

```bash
sudo reposync --newest-only --gpgcheck -l --repoid=${repo} --download_path=${rootpath}
sudo createrepo --update -v ${rootpath}/${repo}
```

On Rocky 8 and later, `--download-metadata` handles that, so the `createrepo` line goes away for a straight mirror. Drop `--newest-only` at the same time, for the reason covered above. You will still want `createrepo_c` if you change what is in the local repository after syncing, such as keeping older RPMs that upstream has dropped (see [Delete old packages on sync](#delete-old-packages-on-sync)).

### Automate reposync

There are several ways to automate fetching the latest packages via `reposync`.

For a complete, centralized solution, we recommend running a playbook within [Ascender](https://ciq.com/products/ascender/).

Alternatively, you can set up a simple [cronjob](https://docs.rockylinux.org/guides/automation/cron_jobs_howto/) to pull the latest packages automatically. For example, you might create `/etc/cron.d/reposync` in the example provided:

```bash
# Run the hourly jobs
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
@daily root dnf -c /root/rocky-baseos-9.repo reposync -v -p repodata --download-metadata --repoid=rocky-baseos-9 --download-path=/var/www/html/rocky-baseos-9/
```

This will run the cronjob daily at midnight. Add a new line for each repository you intend to sync.

For more granular control of `cronjob` timing, please consult [this documentation on cron jobs](https://docs.rockylinux.org/guides/automation/cron_jobs_howto#more-complex-options).

### Delete old packages on sync

When running `reposync`, older packages will not be deleted by default.

This means your local repository will contain all versions of packages, as newer versions are released.

This is helpful if you desire having older packages available that are not retained in the upstream repository. However, this can take up significant storage space.

To delete packages locally that are no longer present on the remote server, append `--delete` to the `reposync` command like so:

```bash
dnf -c ./rocky-baseos-9.repo reposync -v -p repodata \
--download-metadata --repoid=rocky-baseos-9 \
--download-path=/var/www/html/rocky-baseos-9/ \
--delete
```

For a baseline you freeze per patch cycle, leave `--delete` off. The package you are most likely to need mid-cycle is the previous build of something that just shipped broken, and once upstream drops it your mirror is the only copy left. `--delete` removes it on the next sync.

Although the old RPM files remain in the repository folders, they won’t be accessible via DNF on remote systems.

This is because the metadata is copied from upstream.

To retain these preexisting RPMs, update the metadata using a tool like [createrepo](https://github.com/rpm-software-management/createrepo).

## Notes

- `reposync` keeps every package version by default. Pass `--delete` to prune packages no longer upstream, but leave it off for a baseline you freeze per patch cycle.
- Client-visible metadata is copied from upstream, so RPMs you keep locally are not served over DNF until you rebuild the metadata with `createrepo_c`.
- Pair `--newest-only` with `--download-metadata` and the mirror breaks: the metadata still lists the older packages that were skipped.

## References & related articles

[How to Set Up a Web Server in Rocky Linux](https://ciq.com/blog/how-to-set-up-a-web-server-in-rocky-linux/)  
[Rocky Linux Cronjob Guide](https://docs.rockylinux.org/guides/automation/cron_jobs_howto/)  
[DNF RepoSync Options](https://dnf-plugins-core.readthedocs.io/en/latest/reposync.html)  
[DNF RepoManage Options](https://dnf-plugins-core.readthedocs.io/en/latest/repomanage.html)


How to Create a Local Mirror Using Reposync with CIQ Depot


## Introduction

IPv6 is enabled by default and fully supported on all current Rocky Linux releases. It is very stable, and in most environments there is no reason to turn it off. There are unique and uncommon scenarios where disabling it is necessary, such as satisfying a security-scan finding or a compliance requirement, or working around an application that misbehaves with IPv6 present. This article covers how to disable it, starting with the least invasive method, and the caveats of a full system-wide disable. It applies to Rocky Linux 8, 9, and 10.

## Problem

A requirement, often a security scanner, calls for IPv6 to be disabled. Turning it off per interface with a sysctl such as `net.ipv6.conf.<interface>.disable_ipv6=1` disables IPv6 on those interfaces but leaves the IPv6 stack loaded and the `::1` loopback present. A scanner that checks whether the IPv6 stack is loaded at all will still flag the host, which is the usual reason a per-interface change "doesn't work."

## Resolution

***⚠️ WARNING*** *Leave IPv6 enabled unless you have a specific, confirmed reason to disable it, and prefer the least invasive method that satisfies that reason. A full system-wide disable removes the `::1` loopback address; applications that bind to `::1`, and some services that expect IPv6 to be available even when they do not use it, can fail, and you may see new SELinux denials. Test critical services before rolling a full disable out fleet-wide.*

### Option 1: Disable IPv6 on specific connections

If you only need IPv6 off on particular interfaces, disable it per connection with NetworkManager. This is the least invasive option and is persistent across reboots:

```bash
sudo nmcli connection modify <connection> ipv6.method disabled
sudo nmcli connection up <connection>
```

List connection names with `nmcli connection show`. This leaves the IPv6 stack loaded and `::1` in place, so it may not satisfy a scan that requires the stack to be fully disabled. If that is your situation, use Option 2.

### Option 2: Disable IPv6 system-wide

To turn the IPv6 stack off entirely, add the `ipv6.disable=1` kernel parameter to every installed kernel and reboot:

```bash
sudo grubby --update-kernel=ALL --args="ipv6.disable=1"
sudo reboot
```

`grubby --update-kernel=ALL` writes the argument into the boot entry of every installed kernel, and the `ALL` keyword also updates `GRUB_CMDLINE_LINUX` in `/etc/default/grub` so kernels installed later inherit it. This is the change that fully removes the IPv6 stack, including `::1`, which is why it satisfies scanners that a per-interface change does not (see the Warning above for the trade-off).

### Verify

After the reboot, confirm the parameter is active and no IPv6 addresses remain:

```bash
grep -o 'ipv6.disable=1' /proc/cmdline
ls /proc/net/if_inet6
```

The first command echoes `ipv6.disable=1`. When the IPv6 stack is disabled the kernel does not create `/proc/net/if_inet6`, so `ls` reports `No such file or directory`.

### Revert

To re-enable IPv6, remove the parameter and reboot:

```bash
sudo grubby --update-kernel=ALL --remove-args="ipv6.disable=1"
sudo reboot
```

## Notes

- The per-interface sysctl approach (`net.ipv6.conf.all.disable_ipv6=1` and `net.ipv6.conf.default.disable_ipv6=1`) disables IPv6 addressing but leaves the stack loaded. Use it when you want IPv6 addresses gone but do not need the stack removed; use the kernel parameter when a scan or policy requires the stack itself to be off.
- This procedure was developed on AMD x86_64 hardware.
- Before a full disable, check that localhost-dependent services still work with only the IPv4 `127.0.0.1` loopback. Some software is configured to reach `localhost` over `::1`.

## References & related articles

[nm-settings(5) man page](https://rockyman.org/9/NetworkManager/man5/nm-settings.5.html)  
[grubby(8) man page](https://rockyman.org/9/grubby/man8/grubby.8.html)


Disabling IPv6 on Rocky Linux


## Introduction

Many businesses need to have multiple desktop environments set up and to tweak them accordingly as per their requirements.

CIQ delved deep into one such test case, in order to have the MATE desktop environment installed alongside the GNOME desktop environment and to remove the ability to log in to MATE from the GNOME Display Manager (GDM) login screen.

This article will describe how to remove the option to log in to MATE from GDM.

## Problem

An administrator would like to disable the option to log in to MATE or other alternative desktops from the GDM login window without uninstalling the desktop environment.

## Resolution

Simply move the `mate.desktop` file to another location or rename it to `mate.desktop.disabled`:

```bash
mv /usr/share/xsessions/mate.desktop /usr/share/xsessions/mate.desktop.disabled
```

After a reboot, you should no longer see the option at the GDM login screen to log in to MATE when clicking the cog icon after entering your username.

## Conclusion

Following the steps listed in this article by renaming the `mate.desktop` file should help you better control the desktop environments (in this case with MATE) that users have access to from the GNOME Display Manager in your environment.

An example of where this is beneficial is with separating users into two groups: those needing an alternative desktop for specific applications and those that do not and can complete their assigned tasks using GNOME.

## References & related articles

[Rocky Linux Desktops](https://docs.rockylinux.org/desktop/)


How to Remove Access to the MATE Desktop Environment From the GNOME Desktop Manager Login Screen


## Introduction

This article provides a step-by-step guide to manually disable the screensaver lock screen setting on the GNOME desktop environment. Follow the instructions below to complete the process using `dconf` and `dconf-editor`.

## Problem

Users may need to disable the screensaver lock screen setting on their GNOME desktop environment for various reasons, such as preventing interruption during tasks.

## Prerequisites

Make sure your system includes the `dconf` and `dconf-editor` packages. If you need to install them, run the following command:

```bash
sudo dnf install -y dconf dconf-editor
```

## Instructions

### Steps to disable the screensaver lock-screen setting

1. **Launch dconf-editor**

   Open the `dconf-editor` by running the following command in your terminal:

   ```bash
   dconf-editor
   ```

   This action launches the `dconf` GUI.

2. **Navigate to the screensaver settings**

   In the `dconf-editor`, navigate to the screensaver settings by following this path:

   ```text
   org > gnome > desktop > screensaver
   ```

3. **Edit lock screen setting**

   - Click on `lock-enabled`. This opens a screen where you can change the value.
   - Toggle the `Use default value` to off.
   - Change the `Custom value` to `false`.
   - Click the blue check-mark at the bottom to confirm your changes.

   ![dconf-editor screensaver settings](/images/articles/disable-screensaver-lock-screen/screensaver-dconf-editor_m.jpg)

4. **Handling errors**

   If you encounter the following error in your terminal while attempting to save the settings:

   ```text
   (dconf-editor:17050): dconf-WARNING **: [time]: failed to commit changes to dconf: Failed to execute child process "dbus-launch" (No such file or directory)
   ```

   This indicates that the `dbus-launch` utility is missing. Resolve this by exiting the GUI and installing the `dbus-x11` package:

   ```bash
   sudo dnf install -y dbus-x11
   ```

   After installing, relaunch the GUI and attempt to save your changes again. If the error persists, reboot the system.

### Validate the changes

Verify that the lock screen is disabled by running one of the following commands:

```bash
gsettings get org.gnome.desktop.screensaver lock-enabled
```

or

```bash
dconf read /org/gnome/desktop/screensaver/lock-enabled
```

These commands should return `false`, indicating that the lock screen has been disabled.

## Conclusion

Follow these steps to manually disable the screensaver lock screen setting on your GNOME desktop environment. If you encounter any issues, verify that you have installed and correctly configured all required packages.


Disabling the Screensaver Lock Screen Setting


## Introduction

`dracut` builds the initramfs and compresses it with `gzip` by default. It can use other compressors instead (`lz4`, `zstd`, `xz`, and others), set through a `dracut.conf.d` drop-in. In almost all cases we recommend keeping the default `gzip`, which is the most broadly compatible option, and changing the compressor only if you have a specific, tested reason. This article shows how to change it, and what to confirm first, on Rocky Linux 8, 9, and 10.

## Problem

You want to change the initramfs compressor from the default `gzip`, for example to shorten a workflow that regenerates the initramfs repeatedly.

## Resolution

***⚠️ WARNING*** *We recommend leaving the compressor at the default `gzip` unless you have a specific, tested reason to change it. `gzip` is the most broadly compatible choice: every supported Rocky Linux kernel, including rescue and recovery kernels, can decompress it, and it needs no additional package. Other compressors only work when the matching tool is installed and the kernel can decompress that format at boot. Get either wrong and the affected node may fail to boot.*

Set the compressor in a drop-in under `/etc/dracut.conf.d/` rather than editing the package-owned `/etc/dracut.conf`. Files in `/etc/dracut.conf.d/` are read in alphanumeric order and override `/etc/dracut.conf`.

Create a drop-in with the `compress=` directive:

```ini
# /etc/dracut.conf.d/90-compress.conf
compress="lz4"
```

Per `dracut.conf(5)`, `compress=` accepts `gzip` (the default), `bzip2`, `lzma`, `xz`, `lzo` (this is `lzop`), `lz4`, and `zstd`.

Before regenerating, make sure both of these are true:

- **The compressor's tool is installed** on the system building the initramfs. `lz4`, `zstd`, and `lzop` come from packages of the same name; if the tool is missing, the build fails.
- **The kernel can decompress the format at boot.** Each format has a matching config symbol; confirm it is enabled, and check the rescue kernel too:

  ```bash
  grep -E 'CONFIG_RD_(GZIP|BZIP2|LZMA|XZ|LZO|LZ4|ZSTD)=' /boot/config-"$(uname -r)"
  ```

  Find the row for your chosen format; a value of `=y` means the kernel supports it.

Then regenerate the initramfs for the installed kernels so the change takes effect:

```bash
sudo dracut --force --regenerate-all
```

## Notes

- If you set the compressor through configuration management, apply it to one node and confirm it boots before rolling it out fleet-wide.
- Command-line compression flags (`--gzip`, `--lz4`, `--zstd`, and so on) override the drop-in, which is useful for a one-off build without changing the configuration.

## References & related articles

The `compress=` directive and the compression command-line options are documented in the `dracut.conf` and `dracut` man pages (also available on any Rocky Linux system with `man dracut.conf` and `man dracut`).

[dracut.conf(5) man page](https://rockyman.org/9/dracut/man5/dracut.conf.5.html)  
[dracut(8) man page](https://rockyman.org/9/dracut/man8/dracut.8.html)


Changing dracut initramfs Compression on Rocky Linux


## Introduction

`kdump` is a kernel crash dumping mechanism that allows you to save the contents of system memory when a kernel panic or system crash occurs. This memory dump (vmcore) can be analyzed to determine the root cause of the crash. This guide covers installation, configuration, and testing of `kdump` on Rocky Linux 8, 9, and 10, including important differences between versions.

## Problem

System administrators need to capture kernel crash dumps for troubleshooting kernel panics and system failures. Without `kdump` properly configured, diagnostic information is lost when the system crashes, making root cause analysis difficult or impossible.

## Resolution

### Understanding Version Differences

Before configuring `kdump`, be aware of key differences across Rocky Linux versions:

**Rocky Linux 8:**

- Supports `crashkernel=auto` parameter for automatic memory allocation
- The `crashkernel=auto` parameter will only work when the available memory is greater than 1GB

**Rocky Linux 9:**

- The `crashkernel=auto` parameter is officially deprecated, but still available if needed

**Rocky Linux 10:**

Rocky Linux 10 follows the same crashkernel configuration approach as Rocky Linux 9. Use explicit memory range specifications for the `crashkernel` parameter, as `crashkernel=auto` remains deprecated.
### Step 1: Install `kdump` Packages

Install the required `kexec-tools` package:

```bash
sudo dnf install kexec-tools
```

Verify installation:

```bash
rpm -q kexec-tools
```

### Step 2: Configure crashkernel memory reservation

The crashkernel parameter reserves memory for the crash kernel. Configuration differs by Rocky Linux version.

#### For Rocky Linux 8

Check if crashkernel is already configured:

```bash
grep crashkernel /proc/cmdline
```

If using automatic allocation:

```bash
sudo grubby --update-kernel=ALL --args="crashkernel=auto"
```

Or specify a fixed amount:

```bash
sudo grubby --update-kernel=ALL --args="crashkernel=256M"
```

#### For Rocky Linux 9 and 10

Check current configuration:

```bash
grep crashkernel /proc/cmdline
```

Use `kdumpctl estimate` to calculate recommended memory:

```bash
sudo kdumpctl estimate
```

Example output:

```text
Reserved crashkernel:    256M
Recommended crashkernel: 192M

Kernel image size:   57M
Kernel modules size: 8M
Initramfs size:      36M
Runtime reservation: 64M
Large modules:
    xfs: 2686976
    kvm: 1404928
```

Configure the crashkernel based on the recommendation using a fixed value:

```bash
sudo grubby --update-kernel=ALL --args="crashkernel=192M"
```

Alternative: Range-based allocation (for deployments across multiple systems)

If you're deploying the same configuration to multiple servers with different RAM amounts, use a range-based allocation:

```bash
sudo grubby --update-kernel=ALL --args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
```

The kernel automatically selects the appropriate reservation based on total system RAM at boot time:

- If system has 1-4GB total RAM: reserves 192MB for crashkernel
- If system has 4-64GB total RAM: reserves 256MB for crashkernel
- If system has 64GB+ total RAM: reserves 512MB for crashkernel

One crashkernel memory value is reserved per system - the kernel picks the matching range. This is useful for gold images, Warewulf nodes, or automated deployments where the same kernel configuration is used across different hardware.

#### Verify Bootloader Configuration

Check that the parameter was added:

```bash
sudo grubby --info=ALL | grep crashkernel
```

The `grubby` tool directly modifies the Boot Loader Specification (BLS) entries in `/boot/loader/entries/`. No additional steps are needed - the changes will take effect after reboot.

### Step 3: Configure `kdump` target and behavior

Edit the `kdump` configuration file:

```bash
sudo vi /etc/kdump.conf
```

#### Configure Dump Location

The default location is `/var/crash`. To change it, modify the `path` directive:

```bash
path /var/crash
```

For network storage (disabled by default):

```bash
#nfs [2001:db8::1:2:3:4]:/export/tmp
```

For an SSH target:

```bash
#ssh user@my.server.com
#ssh user@2001:db8::1:2:3:4
#sshkey /root/.ssh/kdump_id_rsa
```

#### Configure Core Collector

The `core_collector` allows you to modify how the `vmcore` dump is copied from the system. One feature of `core_collector` is that it includes the ability to reduce the size of the `vmcore` dump using `makedumpfile`.

```bash
core_collector makedumpfile -l --message-level 7 -d 31
```

Parameters explained:

- `-l`: Use lzo compression
- `-c`: Use zlib compression (alternative to -l)
- `-d 31`: Exclude certain memory pages to reduce dump size
- `--message-level 7`: Control verbosity of dump process

For SSH targets, add the `-F` flag:

```bash
core_collector makedumpfile -l --message-level 1 -d 31 -F
```

#### Common `kdump.conf` directives

```bash
# Default dump path
path /var/crash

# Core collector with compression
core_collector makedumpfile -l --message-level 7 -d 31

# Dump level (31 excludes free pages, cache pages, etc.)
# Valid values: 1, 2, 4, 8, 16, 31 (combination)

# For encrypted LUKS volumes, additional memory may be required
```

### Step 4: Enable and start `kdump` service

Enable `kdump` to start on boot:

```bash
sudo systemctl enable kdump
```

Start the `kdump` service:

```bash
sudo systemctl start kdump
```

Check service status:

```bash
sudo systemctl status kdump
```

You should see:

```text
● kdump.service - Crash recovery kernel arming
     Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; preset: enabled)
     Active: active (exited) since Wed 2025-11-26 15:09:01 JST; 35min ago
   Main PID: 1115 (code=exited, status=0/SUCCESS)
        CPU: 1.646s

Nov 26 15:09:00 rocky-linux96 systemd[1]: Starting Crash recovery kernel arming...
Nov 26 15:09:01 rocky-linux96 kdumpctl[1133]: kdump: kexec: loaded kdump kernel
Nov 26 15:09:01 rocky-linux96 kdumpctl[1133]: kdump: Starting kdump: [OK]
Nov 26 15:09:01 rocky-linux96 systemd[1]: Finished Crash recovery kernel arming.
```

### Step 5: Reboot to Apply Crashkernel Reservation

The crashkernel memory reservation only takes effect after a reboot:

```bash
sudo reboot
```

After reboot, verify the crash kernel is loaded:

```bash
cat /proc/cmdline | grep crashkernel
```

Check that memory has been reserved:

```bash
cat /sys/kernel/kexec_crash_size
```

This should show a non-zero value (in bytes). For example, 512MB would show as `536870912`.

Verify `kdump` is active:

```bash
sudo kdumpctl status
```

Expected output:

```text
# From Rocky Linux 9
kdump: Kdump is operational
kdump: Notice: No vmcore creation test performed!
```

### Step 6: Test kdump Configuration

Before relying on kdump in production, test that it captures crashes correctly.

**WARNING: This will crash your system. Only perform on test systems or during maintenance windows.**

Enable the kernel SysRq functionality:

```bash
echo 1 | sudo tee /proc/sys/kernel/sysrq
```

Trigger a kernel panic:

```bash
echo c | sudo tee /proc/sysrq-trigger
```

The system will crash immediately, reboot, and save the vmcore dump.

After the system restarts, check for the crash dump:

```bash
ls -lh /var/crash/
```

You should see a timestamped directory containing:

```text
vmcore           - The crash dump file
vmcore-dmesg.txt - Kernel log at time of crash
kexec-dmesg.log  - kdump kernel boot log
```

Verify the vmcore file was created and has a reasonable size:

```bash
du -sh /var/crash/*/vmcore
```

## Notes

- `kdump` requires reserved memory that is unavailable to the running system. This is a small trade-off for crash dump capability.
- For systems with encrypted LUKS volumes, additional memory may be required for `kdump` to function properly.
- The memory reservation configured with `crashkernel=` is allocated at boot time and cannot be changed without a reboot.
- Network-based crash dumps (NFS, SSH) require network connectivity during the crash kernel boot, which may not always be available.
- The dump level parameter (`-d 31`) excludes free pages, cache pages, and other non-essential memory to reduce dump size while retaining critical information.
- If `kdump` fails to start with "Memory for crashkernel is not reserved" error, verify the crashkernel parameter is set correctly and the system has been rebooted.
- On systems with Secure Boot enabled, the crash kernel must be properly signed.

## References

[Rocky Linux: Kernel Crash Dump Analysis](https://kb.ciq.com/article/rocky-linux/rl-analysis-of-vmcore-dump-with-crash)
[Rocky Linux: Kernel Panics Caused by CrowdStrike Falcon](https://kb.ciq.com/article/rocky-linux/rl-kernel-panic-crowdstrike-falcon)


How to Enable and Configure kdump on Rocky Linux


## Introduction

When installing Rocky Linux in a cloud environment, all directories, including `/tmp`, will typically be installed on a single volume. If you try to use a separate partition or volume for `/tmp`, you might encounter an issue where the entire root filesystem becomes read-only after a reboot. This occurs because the `systemd` `tmp.mount` unit is sometimes masked by default.

When a `systemd` unit is masked, it is linked to `/dev/null`, making it impossible to start. This differs from disabling a service, which prevents it from running at startup, but does allow it to be manually started or started by another process or unit.

You want the `tmp.mount` unit to be disabled so it does not start by default. However, you need `systemd` to start the `tmp.mount` service when it runs the `systemd-fstab-generator` at boot, which converts `/etc/fstab` entries into native `systemd` units. If `tmp.mount` is masked, this process fails, causing the system to mount the root directory as read-only.

## Symptoms

After you add an entry for `/tmp` in `/etc/fstab` and then reboot, you may see many errors related to a read-only filesystem. You will likely also see the following error when attempting to create a file:

```bash
touch /var/tmp/test
touch: cannot touch 'var/tmp/test': Read-only file system
```

## Resolution

Telling `systemd` to unmask this unit will allow you to mount `/tmp` according to how `/etc/fstab` is configured:

```bash
sudo systemctl unmask tmp.mount
```

If you get a read-only filesystem error when attempting this, you need to remount the root volume first:

```bash
sudo mount -o remount,rw /
```

Once the filesystem is remounted and you have verified you can write to it, rerun the `unmask` command. After a reboot, your server should start up as expected.

There is no need to enable this unit as `systemd` will dynamically generate it and start it at boot.

## Root Cause

This issue arises because cloud servers are primarily used for network-specific tasks such as web hosting.

As a result, the need for an in-memory `/tmp` directory (which is generated by default when you start `tmp.mount`), is largely negated because the network is significantly slower than the disk that `/tmp` would run on.

Additionally, since memory is a significant factor in the cost of cloud services, Rocky Linux does not rely on memory-based `tmpfs` directories. Moreover, the need for a separate partition or volume for `/tmp` is only necessary in minimal use cases and is unnecessary for the majority of workloads.

## References & related articles

[systemd-fstab-generator (freedesktop.org)](https://www.freedesktop.org/software/systemd/man/latest/systemd-fstab-generator.html)
[Rocky Linux Admin Guide (PDF)](https://rocky-linux.github.io/documentation/RockyLinuxAdminGuide.pdf)


How to Avoid a Read-Only Filesystem When Using a Separate Partition or Volume for /tmp


## Introduction

This guide focuses on troubleshooting issues encountered during or after setting up rsyslog for encrypted log transport on Rocky Linux. It addresses common problems such as missing modules and misconfigured parameters that can prevent proper decryption of incoming logs. Whether you’re building a centralized logging infrastructure or enhancing your lab environment’s security, this guide provides hands-on insights into troubleshooting and correct configuration.

## Problem

When an `rsyslog` collector is configured to receive logs from multiple systems and intended to secure the log transport using encryption, errors indicating required modules are missing or misconfigured can be encountered. This can cause scenarios where logs are received on the collector, but are not being decrypted correctly, halting proper secure communication.

## Symptoms

The `rsyslog` configuration can produce several notable errors:

### Module loading issue

```bash
could not load module 'gtls', errors: trying to load module /usr/lib64/rsyslog/gtls.so: /usr/lib64/rsyslog/gtls.so: cannot open shared object file: No such file or directory
```

### Parsing errors in the configuration file

```bash
error during parsing file /etc/rsyslog.d/01-collector.conf, on or before line 9: parameter 'StreamDriver.AuthMode' not known -- typo in config file? 
error during parsing file /etc/rsyslog.d/01-collector.conf, on or before line 9: parameter 'StreamDriver.Mode' not known -- typo in config file? error during parsing file /etc/rsyslog.d/01-collector.conf, on or before line 9: parameter 'StreamDriver.Name' not known -- typo in config file?
```

Even though logs are being received, they remain encrypted and unprocessed, indicating a configuration or module issue.

## Resolution

Below are the prerequisites and detailed steps to resolve the issue:

### Testing

* Utilize the `root` account or a user with `sudo` privileges for testing.

* Set up two Rocky Linux 8.10 machines: one designated as the server and the other as the client.

* Set the hostname for each node with an FQDN:

  * Server: `hostnamectl set-hostname rocky-linux810-server.rsyslog.test`

  * Client: `hostnamectl set-hostname rocky-linux810-client.rsyslog.test`

* Update the `/etc/hosts` file on both nodes with the respective IP addresses and hostnames. An example is below (please replace the IPs and hostnames with your own):

```bash
cat << "EOF" | tee /etc/hosts
10.0.0.4    rocky-linux810-server.rsyslog.test  rocky-linux810-server
10.0.0.3    rocky-linux810-client.rsyslog.test  rocky-linux810-client
EOF
```

### Package installation

On both the server and client, install the necessary packages using the following command:
  
```bash
dnf install -y rsyslog rsyslog-gnutls rsyslog-openssl logrotate
```

### Firewall and SELinux adjustments

* Open port `6514/tcp` on both nodes:
  
```bash
firewall-cmd --add-port=6514/tcp --permanent
firewall-cmd --reload
```

* On the server, allow port `6514` through SELinux:

```bash
semanage port -m -t syslogd_port_t -p tcp 6514
```

### Certificate generation

* Create a directory for certificates:
  
```bash
mkdir -p /etc/rsyslog/keys
```

* Generate a self-signed certificate and private key:
  
```bash
openssl req -new -x509 -days 365 -nodes \
  -out /etc/rsyslog/keys/server-cert.pem \
  -keyout /etc/rsyslog/keys/server-key.pem
```

### Directory setup for log storage

* Create directories for logs on both the server and client:

```bash
# Server
mkdir -p /data/logs/rocky-linux810-server

# Client
mkdir -p /data/logs/rocky-linux810-client
```

* Set appropriate permissions for each directory:
  
```bash
# Server
chmod 644 /data/logs/rocky-linux810-server

# Client
chmod 644 /data/logs/rocky-linux810-client
```

* Configure SELinux contexts:
  
```bash
# Server
semanage fcontext -a -t var_log_t "/data/logs/rocky-linux810-server(/.*)?"
restorecon -Rv /data/logs/rocky-linux810-server

# Client
semanage fcontext -a -t var_log_t "/data/logs/rocky-linux810-client(/.*)?"
restorecon -Rv /data/logs/rocky-linux810-client
```

### Correcting the module issue

* The key resolution is to ensure the `gtls` module is correctly referenced. Since the file `/usr/lib64/rsyslog/gtls.so` module is missing, a symbolic link needs to be created:
  
```bash
ln -s /usr/lib64/rsyslog/lmnsd_gtls.so /usr/lib64/rsyslog/gtls.so
```

### Server configuration (`/etc/rsyslog.d/01-collector.conf`)

* Create/update the configuration file with the following content:
  
```bash
cat << "EOF" | tee /etc/rsyslog.d/01-collector.conf
# /etc/rsyslog.d/01-collector.conf

module(load="imtcp")
module(load="gtls")

$DefaultNetstreamDriver gtls

$DefaultNetstreamDriverCAFile   /etc/pki/tls/certs/ca-bundle.crt
$DefaultNetstreamDriverCertFile /etc/rsyslog/keys/server-cert.pem
$DefaultNetstreamDriverKeyFile  /etc/rsyslog/keys/server-key.pem

$InputTCPServerRun 6514

$template RemoteLogs,"/data/logs/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
& ~
EOF
```

### Client configuration (`/etc/rsyslog.d/01-sender.conf`)

* Create/update the configuration file on the client with:
  
```bash
cat << "EOF" | tee /etc/rsyslog.d/01-sender.conf
module(load="omfwd")
module(load="gtls")

$DefaultNetstreamDriverCAFile /etc/pki/tls/certs/ca-bundle.crt

$DefaultNetstreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon

*.* action(type="omfwd"
           target="10.0.0.4"
           port="6514"
           protocol="tcp")
EOF
```

### Service restart and testing

* Restart `rsyslog` on both the server and client:
  
```bash
systemctl restart rsyslog
```

* Test by sending an encrypted message from the client:
  
```bash
logger "THIS IS A TEST ENCRYPTED MESSAGE"
```

* On the server, verify the decrypted log messages:
  
```bash
tail -f /data/logs/rocky-linux810-client/root.log
```

* An example output:
  
```bash
tail -f /data/logs/rocky-linux810-client/root.log
[root@rocky-linux810-server ~]# tail -f /data/logs/rocky-linux810-client/root.log
Apr 30 07:26:07 rocky-linux810-client root[53589]: THIS IS A TEST ENCRYPTED MESSAGE
Apr 30 07:26:07 rocky-linux810-client root[53589]: THIS IS A TEST ENCRYPTED MESSAGE
Apr 30 07:26:08 rocky-linux810-client root[53590]: THIS IS A TEST ENCRYPTED MESSAGE
```

## Root Cause

The core issue stems from a missing symbolic link for the `gtls` module. The `rsyslog` configuration is attempting to load the module from `/usr/lib64/rsyslog/gtls.so`, which does not exist. Instead, the system provides the library as `/usr/lib64/rsyslog/lmnsd_gtls.so`. Creating the symbolic link resolves the `module-not-found` error and allows the proper configuration directives—such as `StreamDriver.Name`, `StreamDriver.Mode`, and `StreamDriver.AuthMode`—to be recognized and processed, thereby enabling successful decryption of incoming logs.

## Notes

* **SELinux and Firewall:** Always ensure that SELinux contexts and firewall rules are appropriately configured, as they can block the secure transport on non-standard ports like `6514`.

* **Module Naming and Configuration:** Module names and parameters can vary between rsyslog releases. It’s advisable to consult the [official rsyslog documentation](https://www.rsyslog.com/doc/index.html) when encountering similar issues.

* **Certificate Management:** For production environments, replace self-signed certificates with those issued by a trusted CA and ensure proper certificate management policies are in place.

## References & Related Articles

[Rsyslog Error 2066 - Missing gtls Module](https://www.rsyslog.com/?s=error+2066)  
[Rsyslog Error 2207 - Unknown Parameter Issues](https://www.rsyslog.com/?s=error+2207)  
[Rsyslog gtls Network Stream Driver](https://www.rsyslog.com/doc/concepts/ns_gtls.html)


Setting Up Rsyslog with Encrypted Transport on Rocky Linux


## Introduction

When working with multiple repositories in Rocky Linux, administrators may wish to control which packages are used from which repository.  

This is particularly useful when specific kernel versions must be preserved or when particular packages must be included from either one repository or another.  

This article outlines how to configure repository definitions to include or exclude packages and avoid unwanted package overrides.  

## Problem

Using multiple repositories with overlapping packages can lead to package conflicts or unintentional upgrades.  

For example, a FIPS-certified repository may install newer kernel packages that override those from an LTS repository if not properly managed.  

Without `exclusions` or `inclusions` in place, the `priority` of a repository alone may not prevent unwanted package selections.  

## Symptoms

Running `dnf update` attempts to install a version of a package from a repository you weren't intending to pull from.  

## Resolution

* The following example uses Rocky Linux 9.2 and the CIQ Portal [`CIQ LTS`](https://ciq.com/services/long-term-support/) for Rocky 9.2 and `CIQ FIPS 9.2 Certified` repositories, but can be applied to any installation of Rocky Linux with more than one repository.

* Install the `depot` RPM:

```bash
dnf install -y https://depot.ciq.com/public/files/depot-client/depot/depot.x86_64.rpm
```

* Log in with your CIQ Depot `Username` and `Access Token`:

```bash
depot login -u <USERNAME> -t <TOKEN>
```

* Enable the necessary repositories using `depot enable`: 

```bash
depot enable lts-9.2  
depot enable fips-9.2-certified  
```

### Excludepkgs example

* In the following example, we are going to update the `kernel*` packages. However, we want to pull and install the packages from the `CIQ LTS for Rocky 9.2` repository and not the `CIQ FIPS 9.2 Certified` repository.

* Modify the FIPS repository configuration file `/etc/yum.repos.d/depot-fips-9.2-certified.repo`.

* In the `[fips-9.2-certified-x86_64]` section, add the following line before `enabled = true`:

```ini
excludepkgs=kernel*
```

* Save the file.

* Once the changes to the file have been made, your repository configuration will look like the below example:

```ini
[fips-9.2-certified-x86_64]  
name = Rocky Linux 9.2 from CIQ - FIPS Certified (x86_64)  
baseurl = https://depot.ciq.com/files/fips-9.2-certified/fips-9.2-certified-x86_64  
gpgkey = https://ciq.com/keys/rpm-gpg-key-ciq  
username = <CIQ_PORTAL_USERNAME_HERE>  
password = <CIQ_PORTAL_PASSWORD_HERE>  
metadata_expire = 5  
priority = 20  
repo_gpgcheck = false  
gpgcheck = true  
excludepkgs=kernel*  
enabled = true  
skip_if_unavailable = true  
```

* Run `dnf update` to verify only LTS kernel packages are selected.

* Optionally use `dnf list --showduplicates` to inspect version availability across repositories. An example is with the `python3-perf` package:

```bash
python3-perf.x86_64                                  5.14.0-284.30.1.el9_2                    rocky-baseos-9.2.x86_64   
python3-perf.x86_64                                  5.14.0-284.30.1.el9_2.ciqfips.0.8.1      fips-9.2-certified-x86_64 
python3-perf.x86_64                                  5.14.0-284.30.1.el9_2.92ciq_lts.6.1      rocky-lts-9.2.x86_64      
python3-perf.x86_64                                  5.14.0-570.22.1.el9_6                    appstream
```

You can use `dnf list` variants to inspect package availability, which helps with debugging repository conflicts. Recommended commands include:

```bash
dnf list --showduplicates
dnf list --available
dnf list --installed
dnf list --obsoletes
dnf list --upgrades
```

Use these commands to understand which package versions are coming from which repositories.

### Includepkgs example

* Using the same repository setup above, `includepkgs` allows us to use particular packages from a repository and ignore all others.

* For example, you are wanting to explore and test the `openssl` package from the `CIQ FIPS 9.2 Certified` repository and don't want to use any of the other packages.

* Open the `/etc/yum.repos.d/depot-fips-9.2-certified.repo` repository.

* In the `[fips-9.2-certified-x86_64]` section, add the following line under `enabled = true`:

```ini
includepkgs=openssl,openssl-libs,openssl-fips-provider,openssl-fips-provider-so
```

* Save the file.

* Your repository configuration will look like this:

```ini
[fips-9.2-certified-x86_64]  
name = Rocky Linux 9.2 from CIQ - FIPS Certified (x86_64)  
baseurl = https://depot.ciq.com/files/fips-9.2-certified/fips-9.2-certified-x86_64  
gpgkey = https://ciq.com/keys/rpm-gpg-key-ciq  
username = <CIQ_PORTAL_USERNAME_HERE>  
password = <CIQ_PORTAL_PASSWORD_HERE>  
metadata_expire = 5  
priority = 20  
repo_gpgcheck = false  
gpgcheck = true 
includepkgs=openssl,openssl-libs,openssl-fips-provider,openssl-fips-provider-so
enabled = true  
skip_if_unavailable = true  
```

* Then when running `dnf update` and filtering by `fips`, you will find that only the above selected packages listed in the `includepkgs` line have been included and no other packages from the FIPS repository:

```bash
[root@rocky-linux92 ~]# dnf update | grep fips
 openssl                          x86_64  1:3.2.2-6.el9_2.ciqfips.1.0.1            fips-9.2-certified-x86_64   1.3 M
 openssl-libs                     x86_64  1:3.2.2-6.el9_2.ciqfips.1.0.1            fips-9.2-certified-x86_64   2.1 M
 openssl-fips-provider            x86_64  3.0.7-27.el9_2.ciqfips.0.2.7             fips-9.2-certified-x86_64   8.7 k
 openssl-fips-provider-so         x86_64  3.0.7-27.el9_2.ciqfips.0.2.7             fips-9.2-certified-x86_64   654 k
```

## Root cause

The FIPS repository has a higher `priority` value (20) than the LTS repository (50), causing it to override package versions.

Without explicitly excluding packages from the FIPS repository, DNF chooses those over the LTS versions.

By using `excludepkgs=kernel*`, DNF is instructed to ignore any kernel packages from the FIPS repository.

## References & related articles

* [dnf man page, in particular the section on dnf list](https://www.man7.org/linux/man-pages/man8/dnf.8.html)


How to Include and Exclude Packages from Repositories on Rocky Linux


## Introduction

The [FIPS](https://docs.ciq.com/rlc/fips/) `openssl` version is built with strict requirements for entropy to provide the necessary security guarantees. It fills up "pools" of truly random numbers when it starts up. If these pools aren't filled, applications that call the `openssl` library get blocked until enough random material is gathered to proceed. Regular `openssl` behaves differently: if it can't get enough true randomness, it falls back to pseudorandomness and continues. It's important to understand that FIPS isn't meant to run in pieces. It's a holistic system. The kernel provides the entropy to the `openssl` library through the `getrandom()` system call.

The [Rocky Linux](https://docs.ciq.com/rlc/) 9 kernel (and upstream 5.10, 5.15, and later kernels) generally provide enough entropy more or less built-in. CentOS 7 (kernel 3.10) and Rocky Linux 8 (kernel 4.18) don't in some cases, and the shortage is most visible on virtualized or headless systems that lack a good hardware entropy source. When the pool runs dry, the FIPS cryptographic path is the first thing to stall, and it can surface in several different ways.

## Problem

The FIPS `openssl` build enforces strict entropy requirements, so on startup and during key operations it draws truly random material from the kernel through `getrandom()`. If the kernel entropy pool has not gathered enough material, that call blocks until more is available. Because FIPS does not degrade to pseudorandomness the way the non-FIPS libraries do, a slow or starved pool surfaces as a hang or a hard failure rather than a slowdown.

Virtualized guests are a common trigger. A VM has no direct access to physical hardware entropy sources, so it depends on the hypervisor and the host to keep its pool topped up. When the host itself is entropy-starved, it cannot feed its guests, and any FIPS cryptographic operations running in those guests can stall.

## Symptoms

The starvation shows up in a few different ways depending on what calls into the FIPS `openssl` library first.

`dnf` metadata downloads time out because the TLS connection cannot complete:

```bash
dnf list
Rocky Linux BaseOS         0.0  B/s |   0  B     17:45
Errors during downloading metadata for repository 'baseos':
 - Curl error (28): Timeout was reached for https://.../BaseOS/x86_64/os/repodata/repomd.xml [SSL connection timeout]
 - Curl error (28): Timeout was reached for https://.../BaseOS/x86_64/os/repodata/repomd.xml [Operation timed out after 99165 milliseconds with 0 out of 0 bytes received]
Error: Failed to download metadata for repo 'baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
```

`sshd` may freeze on startup, preventing remote access to the system.

An application doing TLS fails from the application layer with an OpenSSL `EOF` error, often intermittently or under load, and early in the process's life:

```text
EOF occurred in violation of protocol (_ssl.c:1149)
```

The tell that this is entropy and not a network or certificate problem is where the process is blocked. Attaching `strace` shows it stuck in `getrandom()` rather than in a network or socket call:

```bash
strace -f -p <PID>
```

You can confirm the pool is low by reading the available entropy on the affected system. A healthy pool is typically in the thousands; a value that sits in the low hundreds or keeps dropping under load points to starvation:

```bash
cat /proc/sys/kernel/random/entropy_avail
```

## Resolution

Install `rng-tools` and enable the `rngd` daemon so it feeds extra, high-quality entropy into the kernel pool, preventing it from becoming exhausted.

On a Rocky Linux 8 system:

```bash
dnf install rng-tools
systemctl enable --now rngd
```

Apply the fix on the layer that is actually starved. If a hypervisor host is failing to supply entropy to its guests, enabling `rngd` on the host resolves the problem for every VM it hosts. Likewise, if you run containers with the FIPS `openssl` library installed, enable `rngd` on the host, not inside the container. On a CentOS 7 host:

```bash
yum install rng-tools
systemctl enable --now rngd
```

After `rngd` is running, restart the affected application or service and confirm that the TLS handshake completes and that `entropy_avail` stays healthy under load.

## Root Cause

FIPS-compliant cryptography imposes stricter entropy requirements than the default configuration, and it blocks waiting for the kernel pool rather than falling back to pseudorandomness. The FIPS `openssl` library called `getrandom()`, the kernel could not satisfy the request because the pool was depleted, and the call blocked. To whatever was calling in, that stall surfaced as a `dnf`/TLS timeout, a frozen `sshd`, or an `EOF occurred in violation of protocol (_ssl.c:1149)` error.

On virtualized systems the shortage originates at the virtualization layer: the guest has no direct hardware entropy source and depends on a host that is not supplying enough. That is also why the non-FIPS libraries do not fail in the same conditions, because they degrade to pseudorandomness instead of blocking. Running `rngd` restores a steady supply of entropy and clears the failure.

## Notes

Newer Rocky Linux kernels generally provide adequate entropy without extra tuning, so this problem is most common on older kernels (CentOS 7 and Rocky Linux 8) and in virtualized or headless environments that lack a good hardware entropy source. If you run FIPS workloads on VMs, treat entropy provisioning as part of the build: enable `rngd` on the guests, and make sure the hypervisor host is not itself starved.

## References & related articles

[man Pages - rngd](https://linux.die.net/man/8/rngd)  
[man Pages - random](https://linux.die.net/man/4/random)


Configuring Entropy on CentOS7/RockyLinux8 for FIPS OpenSSL


## Introduction

Generating a comprehensive diagnostic report using the `sosreport` utility is an essential step in troubleshooting and diagnosing issues on Rocky Linux systems.

The `sosreport` gathers critical information such as system logs, hardware details, installed software, and configurations, helping [CIQ's Support Team](https://portal.ciq.com) to rapidly analyze and resolve issues.

This article covers various methods for creating an `sosreport` on Rocky Linux 8.10, Rocky Linux 9.6, and Rocky Linux 10.

## Problem

Users may encounter system instability, performance issues, hardware failures, or other problems that require detailed analysis of the entire system.

## Symptoms

Common symptoms requiring an `sosreport` include system crashes, performance degradation, unexplained errors in logs, failing applications or hardware operations, and general instability.

## Resolution

### Prerequisites

* An installation of Rocky Linux 8, 9, or 10.

* Root access or equivalent `sudo` privileges

* Install the `sos` package with `sudo dnf install -y sos` (in the above three Rocky Linux versions, the package is installed by default).

### Ways to generate an sosreport

#### Basic `sosreport` generation:

```bash
sudo sos report
```

#### Creating sosreport silently with default answers (you will not be prompted for a case ID):

```bash
sudo sos report --quiet
```

#### Generating a report in a custom location:

```bash
sudo sos report --tmp-dir=/path/to/custom/location
```

#### Generating an `sosreport` without being prompted for input:

```bash
sudo sos report --batch
```

#### Specifying a case ID or support reference:

```bash
sudo sos report --case-id=<case-number>
```

#### Limiting modules included in the report:

```bash
sudo sos report -o kernel,networking,system
```

#### Enable all available plugins when generating a report:

```bash
sudo sos report --alloptions
```

### Obfuscating an sosreport

In order to perform obfuscation of an `sosreport`, select one of the above commands and add the `--clean` flag to obfuscate the information:

```bash
sudo sos report --alloptions --clean
```

An obfuscated report would look like the following example:

```bash
sosreport-host0-8-2025-06-20-blzcdrz-obfuscated.tar.xz
```

You can even obfuscate a previously generated `sosreport` using this command:

```bash
sudo sos clean /var/tmp/<YOUR_SOS_REPORT>
```

### Encrypting an sosreport with gpg passphrase encryption

When generating an `sosreport`, run the following command to encrypt it with GPG encryption:

```bash
sudo sos report --encrypt-pass <YOUR_PASSWORD_HERE>
```

To decrypt the archive, run this command and enter the password that was set in the previous step:

```bash
sudo gpg --output /path/to/<SOSREPORT_NAME>.tar.gz --decrypt /var/tmp/<GENERATED_SOSREPORT>.tar.xz.gpg
```

### Generated sosreport location

You will find the location for all `sosreports` available under:

```bash
/var/tmp/sosreport-<HOSTNAME>-<YYYY-MM-DD>-<UNIQUE_STRING_TO_AVOID_REPORT_OVERWRITING>.tar.xz
```

## Root cause

The need for generating an `sosreport` comes from issues such as hardware incompatibility, software misconfiguration, kernel bugs, driver issues, or other system-level issues.

## Notes

Ensure adequate disk space on the target system, as `sosreports` can be large depending on the chosen modules and system state.

**_⚠️ WARNING_** _Sensitive data should be reviewed or sanitized from generated reports before being shared with any vendor._

## References & related articles

[sosreport GitHub Repository](https://github.com/sosreport/sos)


Generating sosreports on Rocky Linux


## Introduction

Google Cloud Platform Rocky Linux VMs experience hostname behavior changes where the system hostname reverts from FQDN to short name after reboot or NetworkManager restart. This issue is specific to versions running on GCP using the official `rocky-linux-8-optimized-gcp` image.

## Problem

When setting a hostname to an FQDN using `hostnamectl set-hostname test.domain.com`, the hostname correctly shows the FQDN initially. However, after a reboot or NetworkManager restart, the hostname reverts to just the short name portion.

## Symptoms

The following behavior is observed:

```bash
# hostnamectl set-hostname test.domain.com
# hostname
test.domain.com
# reboot
# hostname
test
```

NetworkManager logs show hostname changes occurring:

```text
NetworkManager: hostname: static hostname changed from "test.domain.com" to "test"
NetworkManager: policy: set-hostname: set hostname to 'test' (from system configuration)
```

Note that `hostname -f` still returns the FQDN correctly, but the basic `hostname` command only returns the short name.

## Resolution

This issue is caused by a [change](https://github.com/GoogleCloudPlatform/guest-configs/commit/01a11512fb3d674faa47a3001421c8fd72694716) in Google's guest configuration scripts. This new NetworkManager dispatch script is located at:

```text
/etc/NetworkManager/dispatcher.d/google_hostname.sh
```

The dispatcher script contains the following:

```bash
# Ensure that the hostname and IP address are set only for the primary NIC.
new_ip_address=$(jq -r .networkInterfaces[0].ip <<< $instance) new_host_name=$(jq -r .hostname <<< $instance) google_set_hostname
```

`google_set_hostname` is the script that sets the hostname to the shortened, unqualified domain name, version of the current system hostname. The NetworkManager dispatcher is a system service that executes custom scripts in response to network state changes. In this case, the script runs `if [[ $2 == "up" ]];` which runs the `google_set_hostname` any time a network interface comes online.

## References & related articles

[Google Cloud Guest Configs Repository](https://github.com/GoogleCloudPlatform/guest-configs)  
[NetworkManager Dispatcher Documentation](https://networkmanager.dev/docs/api/latest/NetworkManager-dispatcher.html)


Google Cloud Platform Changes Hostname in Rocky Linux


## Introduction

On Rocky Linux 9.4 and later, running `grub2-mkconfig` against efi may fail. A recent grub2 update changed how you should rebuild the GRUB config.

## Problem

When attempting to update the GRUB configuration by directly targeting the EFI path, the command fails indicating the file should not be overwritten.

This occurs when running:

````bash
sudo grub2-mkconfig -o /boot/efi/EFI/rocky/grub.cfg
````

The command will output the following error, and the configuration will not be updated:

````text
Running `grub2-mkconfig -o /boot/efi/EFI/rocky/grub.cfg' will overwrite the GRUB wrapper.
Please run `grub2-mkconfig -o /boot/grub2/grub.cfg' instead to update grub.cfg.
GRUB configuration file was not updated.
````

## Resolution

To correctly update the GRUB configuration on Rocky Linux 9.4 and newer, you must generate the configuration file in the `/boot/grub2/` directory instead of the `/boot/efi/EFI/rocky/` directory.

Run the following:

````bash
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
````

This will successfully generate and update the main GRUB configuration file. The EFI stub will automatically load this configuration on boot.

## Root Cause

On modern EFI systems, the file at `/boot/efi/EFI/rocky/grub.cfg` is no longer the main configuration file. Instead, it acts as a simple "stub" or "wrapper" that contains minimal code to load the actual configuration from `/boot/grub2/grub.cfg`.

This change prevents users from overwriting the critical EFI stub file, which could break the boot process, particularly on systems where that stub is signed for Secure Boot. The error message is an intentional safeguard to enforce the new, safer procedure for updating GRUB configuration.

## Related Articles

[Rocky Linux System Startup Guide](https://docs.rockylinux.org/books/admin_guide/10-boot/)


Grub2 EFI Update Error


## Introduction

EPEL [has recently switched](https://docs.fedoraproject.org/en-US/epel/branches/#_epel_10) to using separate dnf repositories and dist-git branches for all versions of Enterprise Linux 10, after the release of Enterprise Linux 10.1. [This also prompted EPEL](https://forums.rockylinux.org/t/rocky-9-7-rocky-10-1-and-epel/19879) for Enterprise Linux 9 to be moved to EPEL 9.7

One impact of this change, is that some packages such as `gdal-devel` were incorporated directly into a point release and made no longer available in older minor versions of EPEL. Users on lower minor releases of Enterprise Linux (Rocky Linux 9.6 for example), were therefore unable to pull certain packages that they needed from EPEL.

## Problem

Users on CIQ's [RLC Pro](https://ciq.com/services/long-term-support/) product line are unable to acquire the packages that they need from EPEL.

## Symptoms

For example, you are running Rocky Linux 9.6 and have the EPEL repository enabled. You attempt to install the `liblas-devel` package and observe the following error:

```console
[root@rocky-linux96 ~]# dnf install liblas-devel
# Output truncated
Error: 
 Problem: conflicting requests
  - nothing provides gdal-devel needed by liblas-devel-1.8.2-0.3.gitded4637.el9.x86_64 from epel
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
```

## Resolution

To keep getting access to EPEL packages, please utilize the EPEL archive by performing the following steps.

Modify `/etc/yum.repos.d/epel.repo` and point the `epel` repository to the EPEL archive. The below example fixes the above error as shown in the `Symptoms` section:

```bash
sed -i 's|#baseurl=https://download.example/pub/epel/.*/Everything/$basearch/|baseurl=https://dl.fedoraproject.org/pub/archive/epel/9.6/Everything/$basearch/|' /etc/yum.repos.d/epel.repo
sed -i '/^\[epel\]$/,/^\[/{s|^metalink=|#metalink=|}' /etc/yum.repos.d/epel.repo```
```

Observe that the `liblas-devel` package can now be installed successfully:

```console
[root@rocky-linux96 ~]# dnf install liblas-devel
# Output truncated
Installing:
liblas-devel x86_64 1.8.2-0.3.gitded4637.el9 epel 112 k
```

## Root Cause

Some EPEL packages being removed from EPEL and then incorporated into later Enterprise Linux minor version releases, breaks packages on older Enterprise Linux minor versions that are reliant on EPEL.
# Notes
- Pinning your EPEL repository to a specific archive version will prevent you from receiving security updates and bug fixes for EPEL packages.

## References & related articles

[EPEL Branches - Fedora Docs](https://docs.fedoraproject.org/en-US/epel/branches/#_epel_10)  
[Rocky 9.7, Rocky 10.1 and EPEL - Rocky Linux Forum](https://forums.rockylinux.org/t/rocky-9-7-rocky-10-1-and-epel/19879)



How to Access the EPEL Archive on Rocky Linux


## Introduction

ICMP (Internet Control Message Protocol) is a fundamental part of networking, commonly used for diagnostic tools like `ping`. It is a set of rules used by devices to report data transmission errors within a network. During message exchanges between a sender and receiver, unexpected errors may occur, such as messages being too long or data packets arriving out of order, preventing proper assembly. In these instances, the receiver uses ICMP to notify the sender of the error and requests the message to be resent.

## Problem

Limiting the rate of ICMP echo replies is an important security measure to mitigate the risk of ICMP-based attacks, such as ICMP flood attacks. While the `tc` (Traffic Control) command is commonly used to manage traffic flow on Linux, including rate-limiting ICMP replies, an alternative approach is to use iptables through Firewalld's [direct](https://firewalld.org/documentation/direct/) rules.

## Resolution

### Create a new rule

The appropriate limits depend on your security policy and environment, but a good starting point is a rate of 1 reply per second with a burst limit of 5. This allows the server to respond to up to 5 pings in quick succession before enforcing the 1-per-second restriction. The burst setting also helps accommodate short spikes in legitimate traffic:

```shell
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 10 -p icmp --icmp-type echo-request -j DROP
firewall-cmd --reload
```

The `0` and `10` in these rules specify their priority within firewalld. Firewalld processes rules in ascending order of priority, so the first rule with priority 0 will be evaluated before the second rule with priority 10. If traffic matches the first rule meaning its within the rated limit, it will be accepted and will not proceed to the second rule. If the traffic does not match the first rule meaning it exceeds the rate limit, it will advance to the second rule and will be dropped.

### Replace the existing rule

If you need to modify this rule, you will need to delete the rule and recreate it. To see your current list of rules:

```shell
[root@rockylinux ~]# firewall-cmd --direct --get-all-rules

ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
ipv4 filter INPUT 10 -p icmp --icmp-type echo-request -j DROP
```

Now, append the rule you want to delete to `firewall-cmd --direct --permanent --remove-rule`. When adding a new rule, ensure that your priority is set lower than the priority of the `DROP` rule. In our case we will keep with the same priority of `0`:

```shell
firewall-cmd --direct --permanent --remove-rule ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 2/sec --limit-burst 7 -j ACCEPT
firewall-cmd --reload
```

### Delete the rules

To remove these rules completely, we can use the same `--remove-rule` flag as mentioned above. First start by viewing the existing list of rules as before:

```shell
[root@rockylinux ~]# firewall-cmd --direct --get-all-rules

ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
ipv4 filter INPUT 10 -p icmp --icmp-type echo-request -j DROP
```

Now, append the rules you want to delete to `firewall-cmd --direct --permanent --remove-rule`:

```shell
firewall-cmd --direct --permanent --remove-rule ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
firewall-cmd --direct --permanent --remove-rule ipv4 filter INPUT 10 -p icmp --icmp-type echo-request -j DROP
firewall-cmd --reload
```

## Notes

If you need to create, modify, or delete these rules in an environment where the Firewalld service cannot be started (such as a chrooted environment or in a kickstart), you can use [firewall-offline-cmd](https://firewalld.org/documentation/man-pages/firewall-offline-cmd.html) to modify them.

## References & related articles

[Rocky Linux Beginners Guide to Firewalld](https://docs.rockylinux.org/guides/security/firewalld-beginners/)  
[Firewalld Official Documentation](https://firewalld.org/documentation/)  
[Firewalld Direct Rules Documentation](https://firewalld.org/documentation/direct/)  
[Firewalld Offline Command Documentation](https://firewalld.org/documentation/man-pages/firewall-offline-cmd.html)


Configuring ICMP Rate Limiting in Firewalld


## Introduction

It is inevitable that hardware will fail and there are hundreds of possible scenarios.

This guide will detail troubleshooting steps which can be used to identify hardware errors and how to rectify them.

## Problem

A customer informed CIQ that a node in their production cluster had spontaneously crashed and then rebooted itself.

## Symptoms

Checking `/var/log/messages` from the `sosreport`, there were hundreds of `[Hardware Error]: Hardware error from APEI Generic Hardware Error Source` messages observed.

## Resolution

### Prerequisites

Access to the `root` user or a user with escalated privileges.

### Creating an sosreport

When creating a ticket with [CIQ](https://portal.ciq.com), generating and attaching an `sosreport` to the ticket is very helpful. This allows the support team to start their investigation, without further having to ask the customer to generate one.

Install the `sos` package:

```bash
dnf install -y sos
```

Run the `sos report --batch` command to perform unattended data gather of your system.

Once complete, you will find the `sosreport` under `/var/tmp/sosreport-hostname-yyyy-mm-dd-abcdefg.tar.xz`. An example is below:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# ls -l /var/tmp/ | grep sosreport
-rw-------. 1 root root 11094320 Dec 27 02:33 sosreport-Rocky-Linux-9-5-Test-Machine-2024-12-27-elpmjjq.tar.xz
-rw-r--r--. 1 root root       65 Dec 27 02:33 sosreport-Rocky-Linux-9-5-Test-Machine-2024-12-27-elpmjjq.tar.xz.sha256
```

Please upload the `sosreport` to the ticket.

### Analysis of the issue

When observing behavior such as crashes or random reboots, the best log source is `/var/log/messages` - this records the global messages that are generated from the various services running on your server.

Hardware errors are usually identified by the kernel in the `/var/log/messages` log with a `[Hardware Error]` prefix. These messages come from the ACPI Platform Error Interface.

Using a combination of tools such as `grep`, `awk`, and `uniq` you can quickly find and triage repeating errors. The following in `/var/log/messages` from a customer's `sosreport` shows hundreds of memory-related errors:

```bash
grep "Hardware Error" ./sosreport-<NODE_NAME>-2024-12-24-gwwbeqs/var/log/messages | awk '{print $8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20}' | grep "memory error" | uniq -c
 710 section_type: memory error
```

In addition to when a `Hardware Error` occurs, you will also encounter messages of `Corrected error, no action required`. While the error may be temporarily corrected, if these messages continue, it is a sign of a more serious fault. In that scenario, it is best to contact the hardware vendor and replace the affected part outright.

### Recommendations

In the above memory fault example, the recommendation to the customer was to change out their memory modules and they then contacted their hardware team for further analysis.

## References & related articles

`awk` man page: https://man7.org/linux/man-pages/man1/awk.1p.html

`grep` man page: https://man7.org/linux/man-pages/man1/grep.1.html

`sosreport` man page: https://linux.die.net/man/1/sosreport

`uniq` man page: https://man7.org/linux/man-pages/man1/uniq.1.html


How to Identify Hardware Faults in Rocky Linux


## Introduction

A quick intro to the topic at hand.

In some technical processes, reading from and writing to registers may lead to inconsistencies or invalid values due to the simultaneous execution of read and write operations. This article provides an overview of this issue and offers potential solutions for troubleshooting it.

## Problem

If gathering the PCI information from /sys (sysfs), the data shows PCI link speed and status. In some cases, the LnkSta that is returned indicates the following "16GT/s(downgraded)  x4(downgraded)". While 16GT/s and x4 might be the correct speed and width, the output shows "downgraded".

```console
$ lspci | grep downgr
...
        LnkSta: Speed 8GT/s (downgraded), Width x8 (ok)
        LnkSta: Speed 16GT/s (ok), Width x8 (downgraded)
...
```

### This could match the following criteria

- Inconsistencies in data when comparing multiple reads from the same register
- Observing incorrect or unexpected values that do not align with the expected behavior.
- Frequent occurrences of invalid values (e.g., once or twice a week).

## Resolution

While this is not a resolution to the problem, this could help gathering information that you can bring to your hardware provider.

Below is the lspci code that emits the string: “(downgraded)”. Just so we know what criteria is being used here:

```c
static char *link_compare(int type, int sta, int cap)
{
  if (sta > cap)
    return " (overdriven)";
  if (sta == cap)
    return "";
  if ((type == PCI_EXP_TYPE_ROOT_PORT) || (type == PCI_EXP_TYPE_DOWNSTREAM) ||
      (type == PCI_EXP_TYPE_PCIE_BRIDGE))
    return "";
  return " (downgraded)";
}
```

This is called from: cap_express_link()

```c
  w = get_conf_word(d, where + PCI_EXP_LNKSTA);
  sta_speed = w & PCI_EXP_LNKSTA_SPEED;
  sta_width = (w & PCI_EXP_LNKSTA_WIDTH) >> 4;
  printf("\t\tLnkSta:\tSpeed %s%s, Width x%d%s\n",
        link_speed(sta_speed),
        link_compare(type, sta_speed, cap_speed),
        sta_width,
        link_compare(type, sta_width, cap_width));
```

The `link_compare()` code comes from [this discussion:](https://lore.kernel.org/all/20210318171122.GA151712@bjorn-Precision-5520/T/)

In sum, it’s checking the status speed from `PCI_EXP_LNKSTA_SPEED` and if it’s less than cap_speed then it’s always reporting “(downgraded)” for devices other than `PCI_EXP_TYPE_ROOT_PORT`, `PCI_EXP_TYPE_DOWNSTREAM`, `PCI_EXP_TYPE_PCIE_BRIDGE`.

The PCI configuration data, including speed and width information that `lspci` code reads and potentially reports as “downgraded,” is pulled from the [sysfs](https://docs.kernel.org/PCI/sysfs-pci.html) entries under `/sys/bus/pci/devices`. As an example, for NVMe devices, the data is accessed from the corresponding entries for each NVMe device under `/sys/bus/pci/devices/[DEVICE_ID]/config`.

The reading of PCI configuration data is managed by the kernel's PCI subsystem, specifically within the PCI sysfs interface and related functions. The primary functions involved include:

```text
pci_read_config() in drivers/pci/pci-sysfs.c
pci_user_read_config_##size in drivers/pci/access.c
```

These functions handle the reading of configuration data directly from the hardware registers of the PCI devices.

If you use a script that reads these data, is good to understad that it is just getting raw data out of the device on the PCI bus.

For debugging you could try using dmesg and other system logs to check for any hardware events or errors reported by the kernel that might indicate link renegotiation or other PCIe issues.

Look for AER (Advanced Error Reporting) messages which can sometimes provide clues about PCIe link issues:

```bash
dmesg | grep AER
```

## Possible Causes

1. Simultaneous read and write operations on the same register: This can result in capturing an intermediate state instead of the final value, leading to inconsistencies.
2. Incorrect timing between read and write operations: If there's not enough time between reading and writing, it may lead to the capture of an incorrect or invalid value.
3. Hardware malfunctions: Faulty hardware components or software glitches can cause unexpected behavior in register values.

## References & related articles

[lspci: Don't report PCIe link downgrades for downstream ports](https://lore.kernel.org/all/20210318171122.GA151712@bjorn-Precision-5520/T/)  
[Accessing PCI device resources through sysfs](https://docs.kernel.org/PCI/sysfs-pci.html)


Inconsistent Values When Reading and Writing PCI Registers


## Introduction

[CIQ](https://ciq.com/products/rocky-linux/) SIG/Cloud Next is a set of repositories designed to accelerate new cloud features for Rocky Linux, as well as to provide access to proprietary packages that cannot be included in the official Rocky SIG/Cloud due to licensing restrictions. There are currently two repositories: an open-source repository that adds new features, and a Nonfree repository for proprietary software such as drivers for NVIDIA. CIQ SIG/Cloud Next is not intended to replace Rocky SIG/Cloud, but rather to supplement it with proprietary components. You can find the official repository and documentation for the Cloud Next repository on [gitlab](https://gitlab.com/ctrl-iq-public/sig-cloud-next).

## Instructions

We first need to install the appropriate repositories:

### Rocky 9

To install CIQ SIG/Cloud Next on Rocky 9, run the following:

```shell
sudo dnf install https://depot.ciq.com/public/download/ciq-sigcloud-next-9/ciq-sigcloud-next-9.x86_64/Packages/c/ciq-sigcloud-next-release-6-1.el9_5.cld_next.noarch.rpm
```

You can also install CIQ SIG/Cloud Next - Nonfree by running:

```shell
sudo dnf install https://depot.ciq.com/public/download/ciq-sigcloud-next-9/ciq-sigcloud-next-9-nonfree.x86_64/Packages/c/ciq-sigcloud-next-nonfree-release-3-1.el9_5.cld_next.noarch.rpm
```

### Rocky 8

To install CIQ SIG/Cloud Next on Rocky 8, run:

```shell
sudo dnf install https://depot.ciq.com/public/download/ciq-sigcloud-next-8/ciq-sigcloud-next-8.x86_64/Packages/c/ciq-sigcloud-next-release-6-1.el8_10.cld_next.noarch.rpm
```

To install the CIQ SIG/Cloud Next - Nonfree repository, run:

```shell
sudo dnf install https://depot.ciq.com/public/download/ciq-sigcloud-next-8/ciq-sigcloud-next-8-nonfree.x86_64/Packages/c/ciq-sigcloud-next-nonfree-release-3-1.el8_10.cld_next.noarch.rpm
```

### Update the system

Once you have the appropriate repositories installed, you can update your kernel by running `dnf update` or `dnf update kernel`.

### Install Nvidia drivers

You can now also install the NVIDIA Data Center GPU drivers with:

```shell
sudo dnf install nvidia-dc-driver-latest kmod-nvidia-dc-open-latest
```

If you need to choose a specific version of the driver (such as 550 or 570), you can install those directly:

```shell
sudo dnf install nvidia-dc-driver550 kmod-nvidia-dc-open550 --allowerasing
```

This will switch you to the 550 major version, and will not be updated to another major version of the package.

### Mellanox drivers

To make use of the NVIDIA Mellanox DOCA drivers, you'll need to enable the upstream DOCA repo. You can do so by running:

```shell
sudo dnf install doca-repo
```

Once this new repository is installed, you can then install the DOCA packages, including the kernel modules built for Rocky Linux, by installing the following:

```shell
sudo dnf install doca-ofed
```

### Reboot

Once you have the kernel and desired drivers installed, reboot your server.

## Notes

If you already have a kernel installed that is newer than the latest available kernel from CIQ SIG/Cloud Next, you will need to specify which version you would like to install. We first need to look for the available kernels:

```shell
$ sudo dnf --disablerepo="*" --enablerepo="ciq-sigcloud-next" list --showduplicates kernel
Installed Packages
kernel.x86_64       5.14.0-503.14.1.el9_5       @baseos
kernel.x86_64       5.14.0-503.38.1.el9_5       @baseos
Available Packages
kernel.x86_64       5.14.0-503.22.1.el9_5.cld_next.3.1        ciq-sigcloud-next
kernel.x86_64       5.14.0-503.35.1.el9_5.cld_next.2.1        ciq-sigcloud-next
```

In this case we want to install `5.14.0-503.35.1.el9_5.cld_next.2.1`, so we can run the following:

```shell
sudo dnf install kernel-5.14.0-503.35.1.el9_5.cld_next.2.1
```

Once this finishes installing, it should automatically become your default kernel at boot. You can verify this with:

```shell
$ sudo grubby --default-title
Rocky Linux (5.14.0-503.35.1.el9_5.cld_next.2.1.x86_64) 9.5 (Blue Onyx)
```

If this is the incorrect kernel, or you need to rollback to another kernel, you can do so by running the following:

```shell
$ grubby --set-default /boot/vmlinuz-5.14.0-503.35.1.el9_5.cld_next.2.1.x86_64
The default is /boot/loader/entries/0a7500f2dd6247d69028b90889d6d0fb-5.14.0-503.35.1.el9_5.cld_next.2.1.x86_64.conf with index 3 and kernel /boot/vmlinuz-5.14.0-503.35.1.el9_5.cld_next.2.1.x86_64
```

## References & related articles

[CIQ SIG/Cloud Next GitLab Repository](https://gitlab.com/ctrl-iq-public/sig-cloud-next)  
[Rocky Linux SIG/Cloud Documentation](https://sig-cloud.rocky.page)  
[Rocky Linux Kernel Guide](https://docs.rockylinux.org/labs/systems_administration_II/lab7-the_linux_kernel/)


Install the Cloud-Next Kernel in Rocky Linux


## Introduction

Production Rocky Linux systems running CrowdStrike Falcon Sensor may experience kernel panics with general protection faults during container operations. The crashes occur in `__kmem_cache_alloc_node` due to memory corruption originating from the `falcon_lsm_serviceable` kernel module. This article addresses the specific crash signature, diagnostic indicators, and resolution steps for Falcon-induced kernel panics on Rocky Linux 9.

## Problem

Servers running CrowdStrike Falcon Sensor experience kernel panics during container runtime operations, particularly when executing binaries via `runc` or similar container tools. The crash occurs in the kernel's memory allocator (`__kmem_cache_alloc_node`) when Falcon's security hooks intercept system calls. Crash analysis shows the fault originates in Falcon's kernel modules.

## Symptoms

The crash signature is highly specific and identifiable in crash dumps and dmesg logs:

**Primary indicators:**

- General protection fault with non-canonical memory addresses such as `0x289d17010b1fa10a` or `0x60c5387b1ee398de`
- Crash occurs in `__kmem_cache_alloc_node+0x10e/0x2e0` or `__kmalloc`
- Process triggering crash: `runc`, `containerd`, or other container execution tools
- Kernel tainted with `P` (proprietary) and `E` (unsigned module) flags
- Multiple stack frames from Falcon modules: `falcon_lsm_serviceable`, `falcon_lsm_pinned_*`
- Specific Falcon functions in call stack:
  - `cshook_systemcalltable_pre_compat_sys_ioctl`
  - `cshook_security_bprm_check_security`
  - `pinnedhook_security_bprm_check_security`
  - `_ZdlPvmSt11align_val_t` (C++ memory management in kernel space)

**Example crash signature from dmesg:**

```text
general protection fault, probably for non-canonical address 0x289d17010b1fa10a: 0000 [#1] PREEMPT SMP NOPTI
CPU: 9 PID: 3903395 Comm: runc Kdump: loaded Tainted: P E ------- --- 5.14.0-503.33.1.el9_5.x86_64 #1
RIP: 0010:__kmem_cache_alloc_node+0x10e/0x2e0
```

## Resolution

### Step 1: Confirm Falcon as root cause

Extract and examine the crash dump or dmesg output. Look for the specific Falcon crash signature in the call stack:

```text
Call Trace:
 __kmem_cache_alloc_node+0x10e/0x2e0
 __kmalloc+0x4b/0x140
 cshook_systemcalltable_pre_compat_sys_ioctl+0x3b505/0x43340 [falcon_lsm_serviceable]
 cshook_security_bprm_check_security+0x57/0x70 [falcon_lsm_serviceable]
 pinnedhook_security_bprm_check_security+0x39/0x80 [falcon_lsm_pinned_18108]
 security_bprm_check+0x7b/0x100
 do_execveat_common.isra.0+0x1a4/0x280
 __x64_sys_execve+0x3f/0x50
 do_syscall_64+0x5c/0x80
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
```

**Key diagnostic observation:** in typical crashes, Falcon modules account for a significant portion of the call stack such as 23 out of 50 frames, while hardware components like `MegaSAS` and `bnxt_en` show zero appearances in the execution path and no errors in `dmesg`.

To examine crash information from kdump:

```bash
# Check the auto-generated dmesg file
grep -A 30 "Call Trace" /var/crash/*/vmcore-dmesg.txt | grep falcon
```

### Step 2: Rule out hardware components

Verify that hardware components aren't contributing to the crashes:

```bash
# Verify the RAID controller shows no errors
sudo dmesg | grep -i "mega\|raid" | grep -i error

# Verify the network driver shows no errors
sudo dmesg | grep -i "bnxt\|network" | grep -i error

# Check if these components appear in the crash stack
grep -E "megaraid|bnxt_en|mpt3sas" vmcore-dmesg.txt
```

If the hardware components are clean and absent from the crash stack, the issue is isolated to Falcon.

### Step 3: Contact CrowdStrike support

Engage CrowdStrike support with the crash analysis showing Falcon module involvement:

- Provide the `vmcore` dump or extracted `dmesg` logs showing the call stack
- Include the Falcon Sensor version: `sudo /opt/CrowdStrike/falconctl -g --version`
- Include the kernel version: `uname -r`
- Reference the specific functions in the crash: `cshook_systemcalltable_pre_compat_sys_ioctl`, `falcon_lsm_serviceable`
- Request information about known issues with your kernel version and container runtimes

### Step 4: Temporarily disable Falcon for isolation test

On a non-production or isolated affected system, turn off Falcon to definitively confirm if that is the root cause or not:

```bash
# Stop Falcon sensor (this should unload modules automatically)
sudo systemctl stop falcon-sensor

# Disable automatic startup
sudo systemctl disable falcon-sensor

# Verify Falcon modules are unloaded
lsmod | grep falcon

# If modules remain loaded, unload them manually
for module in $(lsmod | grep falcon | awk '{print $1}'); do
    sudo modprobe -r "$module"
done
```

Monitor the system under normal container workload. If kernel panics cease without Falcon loaded, this definitively confirms Falcon as the root cause.

### Step 5: Apply resolution from CrowdStrike

Follow the resolution provided by CrowdStrike support. This may include Sensor updates, configuration changes, or kernel version recommendations based on their compatibility matrix.

### Step 6: Re-enable and monitor

After applying updates, re-enable Falcon and monitor for any recurrence:

```bash
sudo systemctl enable falcon-sensor
sudo systemctl start falcon-sensor

# Verify Falcon modules loaded
lsmod | grep falcon

# Monitor for crashes
sudo journalctl -k -f | grep -i "general protection\|kernel panic"
```

## Notes

- This specific crash pattern occurs when Falcon sensor intercepts container execution (`execve` system calls) during binary launches.
- The crash occurs when Falcon's `cshook_systemcalltable_pre_compat_sys_ioctl` function attempts memory allocation and encounters corrupted pointers.
- Diagnostic evidence includes stack traces showing Falcon module involvement and absence of hardware errors in `dmesg` logs.
- The non-canonical memory addresses indicate pointer corruption within the Falcon module code path.
- This issue affects containerized workloads more frequently than traditional process execution due to the higher rate of `execve` calls in container environments.
- Falcon sensor versions and kernel compatibility should be verified together - not all sensor versions are certified for all kernel versions.

## References & Related Articles

[Rocky Linux: How to Enable and Configure kdump](https://kb.ciq.com/article/rocky-linux/rl-enable-configure-kdump)
[Rocky Linux: Kernel Crash Dump Analysis](https://kb.ciq.com/article/rocky-linux/rl-analysis-of-vmcore-dump-with-crash)


Kernel Panics Caused by CrowdStrike Falcon Sensor on Rocky Linux


## Introduction

This article provides guidance on migrating a Rocky Linux virtual machine created locally using KVM to Azure.

Follow these steps to ensure a smooth migration from your local Rocky Linux KVM-generated VM to Azure.

## Problem

You need to migrate a Rocky Linux virtual machine created in a local KVM environment over to Azure.

## Symptoms

You have a Rocky Linux VM running locally on KVM and require it to run in Azure.

## Resolution

### Prerequisites

#### KVM

* Rocky Linux host with at least a 500GB disk, 32GB of RAM, and 16 vCPUs available.

* `libvirt` installed (please see [Setting up libvirt on Rocky Linux](https://docs.rockylinux.org/guides/virtualization/libvirt-rocky/) on how to install `libvirt` on Rocky Linux.

* Rocky Linux DVD ISO.

#### Azure

* Azure subscription.

* Azure Recovery Services vault.

* Azure Virtual Network.

### Steps to Resolution

#### Azure step 1

In Azure, navigate to `Recovery Services vaults` and create a new vault under your `Resource Group`:

![The "Create Recovery Services vault" "Basics" step from Azure is shown.](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-1.webp)

Then navigate to `Virtual networks`, select `Create`, and select your `Resource Group` and your `Region`.

!["Create virtual network" "Basics" step is displayed.](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-2.webp)

Under your `Recovery Services vault`, select `Enable Site Recovery`, then `VMware machines to Azure`, and after that click `1: Prepare Infrastructure`.

![In the example "Recovery Services Vault", VMware machines to Azure is highlighted and then the link to "1: Prepare infrastructure" is also highlighted beneath that.](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-3.webp)

* Download the `.OVA` image from Azure to your local machine.

#### KVM steps

Set up a new Rocky Linux VM on KVM. Assign it a 16GB disk, at least 2 vCPUs, and a minimum of 4096 MB RAM. `virt-manager` is the recommended option.

Before initiating the installation of Rocky Linux, under `Hardware` settings, change the `Disk bus` for device `VirtIO Disk 1` from `VirtIO` to `SCSI`. This is needed in order for the disk to boot on Azure once the VM is migrated over.

![The Hardware settings of a test Rocky Linux VM in virt-manager is displayed, with "VirtIO Disk 1" selected and the "Disk bus" of "SCSI" chosen in the drop-down menu in the middle of the screen.](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-4.webp)

Apply the changes and install Rocky Linux.

Ensure all necessary firewall ports are open:

```bash
sudo firewall-cmd --add-port=443/tcp --permanent
sudo firewall-cmd --add-port=1024-65535/tcp --permanent
sudo firewall-cmd --add-port=135/tcp --permanent
sudo firewall-cmd --add-port=443/udp --permanent
sudo firewall-cmd --reload
```

After installing Rocky Linux, shut down the VM and remove the `SATA CDROM 1` from VM's hardware list.

![Hardware list in virt-manager for the Rocky Linux VM, highlighting that "SATA CDROM 1" needs to be removed. The "SATA CDROM 1" option is under the "SCSI Disk" option on the left-hand side](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-5.webp)

In the VM's hardware list, select the XML tab (you need to enable XML editing in `virt-manager` beforehand) and modify the XML configuration of the `SCSI Disk 1` exactly as in the following:

```xml
<disk type="file" device="disk">
  <driver name="qemu" type="qcow2"/>
  <source file="/PATH/TO/<YOUR_QCOW2_IMAGE>.qcow2"/>
  <target dev="sda" bus="scsi"/>
  <address type="drive" controller="0" bus="0" target="0" unit="0"/>
</disk>
```

Power up the VM and verify it boots successfully.

#### Azure Site Recovery VM steps

Extract the required `vmdk` image files from the `.ova` image that you downloaded earlier in the Azure steps:

```bash
tar -xvf ./MicrosoftDRAppliance.ova
```

Convert the extracted `.vmdk` images to `.qcow2`:

```bash
qemu-img convert -O qcow2 MicrosoftDRAppliance_disk0.vmdk MicrosoftDRAppliance_disk0.qcow2
qemu-img convert -O qcow2 MicrosoftDRAppliance_disk1.vmdk MicrosoftDRAppliance_disk1.qcow2
```

Create a new VM in `virt-manager` using the `MicrosoftDRAppliance_disk0.qcow2` image, set the OS as `Windows Server 2016`, provision the VM with a minimum of 16GB RAM, 8 vCPUs, and 200GB storage. 

Before starting the VM, attach the `MicrosoftDRAppliance_disk1.qcow2` image as a separate disk.

![Adding the "MicrosoftDRAppliance_disk1.qcow2" image to the Azure Site Recovery VM under "Storage" --> "Manage" and then clicking "Finish".  ](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-6.webp)

Boot the VM and complete the initial setup, sign-in to Azure and provide your `Registration key` that you received from the `1: Prepare Infrastructure` step in Azure.

Under `Configure vCenter details` and `Provide physical server details`, make sure you select the options for a Physical server being migrated and provide the `root` login credentials and IP address for your local Rocky Linux VM.

The process will take about 30 minutes to complete, with a green checkmark shown at the end if it was successful or not.

#### Azure step 2

In Azure Portal, navigate to `Recovery Services vault`, then `Site Recovery`, after that `VMware machines to Azure` and then `Enable replication`.

Under `Machine type`, select `Physical machines`, choose your Rocky Linux VM and click `Next`.

Under `Source settings`, leave everything as default, and set a `Target name` for your Rocky Linux VM. An example is `rocky-linux-kvm-vm`. Once done, click `Next`.

Under `Target properties`, select your `Failover Azure network` and `Cache storage account`, keep everything else as default, and click `Next`.

Click `Enable replication`.

Check the replication status under `Replicated items` in your `Recovery Services vault`.

["Replicated items" in Azure, displaying the Rocky Linux VM in "Enabling protection" "Status" with a "Replication Health" of "Healthy"](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-7.webp)

Once the status of `Enabling protection` has gone, you can then initiate a `Test Failover`.

["Replicated items" highlighting the "Test Failover" option that can be selected by clicking the menu with the three dots on the right-hand side of the screen.](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-8.webp)

Once replication completes, initiate a `Test Failover`.

After successful testing, verify the VM is available and running under `Virtual machines` in Azure.

Select `Cleanup test failover` under `Replicated items` to remove the VM that was generated.

Power down your local KVM Rocky Linux VM.

Under `Replicated items`, choose `Failover`.

Once the `Failover` is complete, go back to your `Replicated items` page and click `Complete Migration`.

If the new Virtual Machine in Azure is working as expected, then press `OK` and Azure will then cleanup all of the configurations and settings that were generated during the Migration process.

Your local KVM VM is now up and working on Azure!

## Root cause

Migration of a KVM-based Rocky Linux VM to Azure requires specific configurations, including correct disk types (SCSI), appropriate firewall configurations, and usage of the Azure Site Recovery Appliance for replication and failover.

## Notes

Ensure proper firewall and disk configuration before migration to prevent connectivity and boot issues.

Always validate the VM in Azure using the test failover feature before committing to a full failover.

## References & related articles

[Azure Site Recovery Overview](https://learn.microsoft.com/en-us/azure/site-recovery/site-recovery-overview)

[Azure Documentation](https://learn.microsoft.com/en-us/azure/?product=popular)


Migrate a Rocky Linux KVM Virtual Machine to Azure


## Introduction

This article provides step-by-step instructions for migrating a local `Rocky Linux 8.10` `KVM` virtual machine to `Microsoft Azure` using `Azure Migrate` and the `Azure Site Recovery` replication appliance.

The migration process involves preparing your source `VM`, setting up the necessary `Azure` infrastructure, configuring the replication appliance, and performing the actual migration.

## Problem

Organizations need to migrate on-premises virtual machines often running on `KVM` to `Microsoft Azure` for cloud adoption, disaster recovery, or infrastructure modernization purposes. The amount of VMs to migrate to the cloud can range in the thousands and services such as `Azure Migrate` help to make the transition smoother.

The migration process requires specific configuration steps for both the source `VM` and `Azure` environment to ensure successful replication and migration.

## Symptoms

You have a fleet of `Rocky Linux` `VM`s running on `KVM` that needs to be migrated to `Azure`.

The `VM` requires migrating to the cloud whilst maintaining its configuration and data integrity.

## Resolution

### Prerequisites

#### Rocky Linux

* A `Rocky Linux 8.10` `KVM` `VM` (as an example, but Rocky Linux 9 and Rocky Linux 10 are also fine) with a minimum of:

* 4096 MB of `RAM`

* 2 `vCPUs`

* 16GB of disk space

* `SCSI` disk type

#### Windows Server

* A `Windows Server 2022` `VM` on `KVM`, also with a minimum of:

* 16GB of `RAM`

* 8 `vCPUs`

* 700GB of disk space (the replication appliance requires at least 600GB of disk space cache available)

*Please check [Azure Appliance requirements](https://learn.microsoft.com/en-us/azure/migrate/migrate-replication-appliance?view=migrate-classic#appliance-requirements)*

#### Azure

* An active `Azure` subscription with appropriate permissions

* Access to the `Azure Portal`

### Step 1: prepare the source Rocky Linux VM

* Deploy a local `Rocky Linux 8.10` `KVM` and upgrade all packages.

* Provide the `VM` with 4096 MB of `RAM`, 2 `vCPUs` and 16GB of disk as listed in the Prerequisites.

* Change the bus type to `SCSI` before installing the `OS`. This is performed under `Hardware` --> `VirtIO Disk 1` --> `Disk bus` and changing the bus type to `SCSI`

* In the `KVM` `VM`, enable the following firewall ports for communication with the replication appliance:

```bash
sudo firewall-cmd --add-port=135/tcp --permanent
sudo firewall-cmd --add-port=443/tcp --permanent
sudo firewall-cmd --add-port=1024-65535/tcp --permanent
sudo firewall-cmd --add-port=443/udp --permanent
sudo firewall-cmd --reload
```

* Shut down the `VM`

* Remove the `SATA CDROM 1` hardware.

* Change the `SCSI Disk 1` `XML` to the following:

```xml
<disk type="file" device="disk">
  <driver name="qemu" type="qcow2"/>
  <source file="/PATH/TO/<YOUR_QCOW2_IMAGE>.qcow2"/>
  <target dev="sda" bus="scsi"/>
  <address type="drive" controller="0" bus="0" target="0" unit="0"/>
</disk>
```

* Power up the `VM` and verify it boots successfully.

### Step 2: prepare the Windows Server 2022 VM for the replication appliance

* Deploy a `Windows Server 2022` `VM` on `KVM` with 16GB of `RAM`, 8 `vCPUs` and 700GB of disk space. Choose the `Standard Desktop` installation.

* Set a password and then login to the Windows Server.

### Step 3: configure the Azure virtual network

* Go to `Virtual Networks` in the `Azure Portal` and click `Create`

* Under `Basics`, select your `Resource Group`, enter a `Virtual network name` and choose your region.

* For `Security`, leave all options at default and click `Next`

* For `IP addresses`, leave all options at default and click `Next`

* In `Tags`, apply your tags and click `Next`

* Click `Create`

### Step 4: set up the Azure Migrate project

* Go to `Azure Migrate` --> `Servers, databases and web apps` and select `Create project`

* Add your `Resource Group`, `Project Name`, `Geography` and click `Create`

* Under `Assessment tools`, go to `Migration and modernization` (**note, do not use the "Azure Migrate: Discovery and assessment" option
**) and click `Discover`

* For `Where do you want to migrate to?`, choose `Azure VM`

* For `Are your machines virtualized?`, select `Physical or other (AWS, GCP, Xen, etc.)`

* Select your `Target region`

* Check the box that says `Confirm that the target region for migration is "<REGION>"`

* Click `Create resources`

* Return to `Discover` and specify the same options for `Where do you want to migrate to?`, `Are your machines virtualized?` and `Target region`

* For `Experience type`, select `Simplified Experience (Recommended)`

* Go to Step 2 and click `Generate key` to create your `ASR replication appliance key`

* Store this key in a safe location for later use in the process.

### Step 5: Install and Configure the Replication Appliance

* Download the installer from the provided link to your `Windows Server 2022` `VM`

* Extract the zip file's contents.

* Open a `PowerShell` window as an Administrator, navigate to the folder and run `.\DRInstaller.ps1`

* If you receive an error about the script not being digitally signed, run this command instead and modify the path for your folder's location:

```text
powershell -ExecutionPolicy Bypass -File "C:\Users\Administrator\Downloads\DRAppliance\DRAppliance\DRInstaller.ps1"
```

* Go to the desktop and open the shortcut to the `Microsoft Azure Appliance Configuration Manager`

* Click `Continue` through the `Set up prerequisites`, `Appliance components` and `Select Replication appliance connectivity` steps.

* Under `Register with Recovery Services vault`, provide a friendly name and the `ASR replication appliance key` you generated earlier.

* Click `Login` and then `Copy code and Login`

* Log in with your `Azure` credentials.

* Once the appliance is registered, click `Continue`

* Under `Configure vCenter details`, check `I do not have vCenter Server/vSphere ESXi server. I'll protect my servers by manually discovering them using IP addresses.` and press `Continue`

* For `Provide physical server credentials`, click `Add credentials`

* Select the `Operating System` as `Linux`, provide a friendly name and input the `root` user's credentials.

* Click `Add`

* Under `Provide physical server information`, click `Add server`

* Input the `IP` address of your `Rocky Linux 8.10` `VM` and click `Add`

* Click `Continue` and wait approximately 30 minutes for the process to complete.

* Verify you see a green checkmark with `Completed appliance configuration successfully.`

### Step 6: configure replication

* In the `Azure Portal`, go to `Azure Migrate: Server Migration` and click `Replicate`

* Leave other options as default and for `Are your machines virtualized?` choose `Physical or other (AWS, GCP, Xen, etc.)`

* From `On-premises appliance`, choose your appliance and select `Continue`

* For `Basics` --> `Guest credentials`, choose your `Rocky Linux 8.10` `VM's` guest credentials.

* Select the checkbox next to your `VM` when it appears and click `Next`

* For `Target settings`, choose your `Virtual network` created earlier, leave other options as default and click `Next`

* At `Compute`, click `Next`

* For `Disks` --> `Disk Type`, select `Standard SSD` (or another disk type of your choice) and click `Next`

* For `Tags`, add your tags and click `Next`

* At `Review + Start replication`, choose `Replicate`

### Step 7: perform the migration

* Skip the `Test migration` step if not needed and go to `Migrate`

* Leave the default `Azure VM` option and click `Continue`

* Select the `VM` you want to migrate and leave `Yes, shutdown virtual machines(Ensures no data loss)` checked.

* Click `Migrate`

### Step 8: configure network access for the migrated VM

* Once complete, your local `VM` will shut down.

* Go to `Virtual Machines` in `Azure Portal` to access your newly migrated `VM`

* Navigate to your newly created `VM` under `Virtual Machines`, then `Networking` and then `Network settings`

* Click `Add network security group`

* When the `network security group` is created, click `Create port rule` and select `Inbound port rule`

* Specify an `IP` address, source and destination ports, the protocol to use, and the `Action` to allow or deny traffic.

* Click `Save`

* For temporary `SSH` access testing, you can set `Source` to `Any`, `Source port ranges` to `*`, `Destination port ranges` to `22`. Please check with your Security Team on how ports, protocols and IPs should be set up first for best security practices.

* Click `Add`

### Step 9: configure a public IP address

* Under `Networking` --> `Network settings`, check that a `Public IP address` is listed.

* If not present, click `(Configure)` and click on your `NIC` listed in the table. An example `NIC` name is `nic-rocky-linux-810-00-ipconfig`

* Check the box next to `Associate public IP address` and click `Save`

* Refresh the `IP configurations` page to see the `Public IP Address`

* Go to your `VM`, then `Connect` and select `Native SSH`

* `ssh` into your machine with the `root` account or another configured account.

## Notes

The migration process may take considerable time depending on the size of your source `VM` and network bandwidth.

Always perform a test migration before executing the final migration in production environments.

## References & related articles

[Set up the appliance through PowerShell](https://learn.microsoft.com/en-us/azure/site-recovery/deploy-vmware-azure-replication-appliance-modernized#set-up-the-appliance-through-powershell)

[Azure Site Recovery Appliance Installer](https://aka.ms/V2ARcmApplianceCreationPowershellZip)


Migrating a Rocky Linux KVM VM to Microsoft Azure


## Introduction

OpenLDAP clients and servers are capable of using the Transport Layer Security (TLS) framework to provide integrity and confidentiality protections and to support LDAP authentication using the SASL EXTERNAL mechanism. When using an unencrypted port for LDAP authentication, clients may be able to authenticate, however it may not be the right setting based on your security policies.

## Problem

When LDAP authentication occurs over an unencrypted port, certificate validation issues may be bypassed leading to a successful authentication. OpenLDAP servers, by default, demand certificate verification, which can cause failures if the certificate is self-signed or untrusted.

## Symptoms

LDAP authentication failures when using the encrypted port and not when using the unencrypted port may produce errors such as:

```text
7: AUTH: Bind failed ß------- This is where the connection breaks
7: AUTH: LDAP ERROR: -1: Can't contact LDAP server
```

## Resolution

To modify the client TLS settings, update the LDAP client configuration. Open the LDAP configuration file in a text editor:

```shell
sudo vi /etc/openldap/ldap.conf
```

Locate the `TLS_REQCERT` directive and modify its value based on your security requirements.

To bypass certificate validation completely (not recommended for secure environments):

```text
TLS_REQCERT never
```

To allow authentication with an untrusted certificate without disabling validation completely:

```text
TLS_REQCERT allow
```

Save the file and exit the editor. Restart any services dependent on LDAP authentication, or reboot the system if necessary.

## Root Cause

This issue arises because OpenLDAP servers, by default, demand certificate verification (`TLS_REQCERT demand`). If the certificate is self-signed or untrusted, authentication fails unless this setting is adjusted.

## Notes

The default OpenLDAP behavior (`TLS_REQCERT demand`) is intended for secure environments. Changing this setting can reduce security.

If modifying `TLS_REQCERT`, ensure other security measures are in place to prevent unauthorized access.

## References & Related Articles

[OpenLDAP TLS documentation](https://www.openldap.org/doc/admin26/tls.html)


OpenLDAP Fails to Authenticate Over Encrypted Port


## Introduction

When a host is enrolled in FreeIPA and configured to source sudo rules through the System Security Services Daemon (SSSD), a user's effective sudo privileges can come from several places at once: rules applied directly to the user, and rules inherited from one or more group memberships. This article shows how to list every sudo rule that applies to a given user from a single command, instead of cross-referencing the FreeIPA web UI by hand.

## Problem

Sudo rules can be applied directly to a user or through one or more group memberships in FreeIPA. There is no single `ipa` command that lists the complete, effective set of sudo rules for a user. Verifying a user's privileges through the FreeIPA web UI therefore requires a lot of clicking, cross-checking each group the user belongs to, and manual effort that is easy to get wrong.

## Resolution

Use the `sudo` command's own listing mode, which queries SSSD (and therefore FreeIPA) for the effective rules. Run the following as root, or prefixed with `sudo`, because querying another user's privileges requires elevated access:

```bash
sudo -ll -U <username>
```

- `-l` (lowercase L) lists allowed (and forbidden) commands.
- A second `-l` (`-ll`) prints the rules in the longer, more readable format, including where each rule comes from.
- `-U <username>` selects the target user to report on, rather than the user running the command.

For each matching rule, the output shows the source as an `LDAP Role`, confirming the rule was delivered from FreeIPA through SSSD:

```text
LDAP Role: sudo_rule
    RunAsUsers: ALL
    RunAsGroups: ALL
    Options: !authenticate
    Commands:
        /bin/su
        /usr/bin/top
        /usr/bin/du
        /usr/bin/df
```

Each `LDAP Role` block is one FreeIPA sudo rule that applies to the user, whether it was assigned directly or inherited through a group. This gives you the complete, effective picture in one place.

## Root Cause

FreeIPA has no dedicated `ipa` subcommand that resolves and lists every sudo rule applied to a user, including rules inherited through group membership. The `sudo -ll -U` command fills this gap because it evaluates the rules exactly as they are enforced on the host.

## Notes

- The host must be enrolled in FreeIPA with SSSD configured to provide sudo rules (the `sudo` service enabled in SSSD and `sudoers: sss` present in `/etc/nsswitch.conf`). If sudo rules are not sourced from SSSD, only local `/etc/sudoers` rules appear.
- SSSD caches sudo rules. If a rule was changed recently in FreeIPA and is not yet reflected, refresh the cache with `sudo sss_cache -E` (or restart `sssd`) and run the command again.


Listing All Sudo Rules for a FreeIPA User


## Introduction

CIQ had a customer issue, where the customer was attempting to set up local users and then authenticate using `Kerberos` to their `Active Directory` server. The following guide will explain how this was resolved.

## Problem

The customer was unable to authenticate via `ssh` to their Rocky Linux 9.2 [LTS](https://ciq.com/services/long-term-support/) node using a local user account stored on the node. No passwords were locally saved in the `/etc/shadow` file and the passwords were only available on the `Active Directory` server.

The customer's configuration of `sssd`, `Kerberos` and `Active Directory` had worked without issue on CentOS 7.9, CentOS 8.1 and Rocky Linux 8.5. It was only on Rocky Linux 9.2 LTS where the customer was facing issues.

## Symptoms

When attempting to log in to the server via `ssh`, the following authentication messages were observed under `/var/log/secure`:

```bash
Dec 9 11:07:31 <server_name> sshd[1155]: Received signal 15; terminating.
Dec 9 11:07:31 <server_name> sshd[79752]: Server listening on 0.0.0.0 port 22.
Dec 9 11:07:31 <server_name> sshd[79752]: Server listening on :: port 22.
Dec 9 11:10:04 <server_name> unix_chkpwd[79934]: check pass; user unknown
Dec 9 11:10:04 <server_name> unix_chkpwd[79934]: password check failed for user (<username>)
Dec 9 11:10:04 <server_name> sshd[79919]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<ip_address> user=<username>
Dec 9 11:10:44 <server_name> sshd[79919]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<ip_address> user=<username>
Dec 9 11:10:44 <server_name> sshd[79919]: pam_sss(sshd:auth): received for user <username>: 4 (System error)
```

Account retrieval errors could also be seen via the `/var/log/sssd/sssd_pam.log` and `/var/log/sssd/sssd_nss.log` files:

```bash
[root@<server_name> ~]# cat /var/log/sssd/sssd_pam.log
(2024-12-16 13:45:58): [pam] [server_setup] (0x3f7c0): Starting with debug level = 0x0070
(2024-12-16 13:47:20): [pam] [cache_req_common_process_dp_reply] (0x3f7c0): [CID#1] CR #3: Could not get account info [<account_number>]: SSSD is offline

[root@<server_name> ~]# cat /var/log/sssd/sssd_nss.log
(2024-12-16 13:45:58): [nss] [server_setup] (0x3f7c0): Starting with debug level = 0x0070
(2024-12-16 13:47:20): [nss] [cache_req_common_process_dp_reply] (0x3f7c0): [CID#5] CR #17: Could not get account info [<account_number>]: SSSD is offline
```

## Resolution

The customer was utilizing `id_provider = files` in their `/etc/sssd/sssd.conf` file. Since the Rocky Linux 9.x series, [this is a deprecated option in sssd](https://sssd.io/docs/files-provider-deprecation.html).

Instead, they added the following settings into their `sssd.conf` file:

```bash
id_provider = proxy
krb5_server = example.com:88
proxy_lib_name = files
```

Then restarted the `sssd` service:

```bash
systemctl restart sssd
```

The customer's `sssd.conf` file ultimately looked like the following (all sensitive customer information has been obfuscated):

```bash
[sssd]
    services = nss, pam
    domains = EXAMPLE.COM
    default_domain_suffix = example.com

[domain/EXAMPLE.COM]
    id_provider = proxy
    proxy_lib_name = files
    auth_provider = krb5
    krb5_realm = EXAMPLE.COM
    krb5_server = example.com:88
    krb5_validate = false
    krb5_ccachedir = /var/tmp
    krb5_keytab = /etc/krb5.keytab

[nss]
filter_groups = root,<local user>
filter_users = root,<local user>
```

In addition, the customer's `/etc/krb5.conf` file:

```bash
includedir /etc/krb5.conf.d/

[logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

[libdefaults]
    default_realm=EXAMPLE.COM
    dns_lookup_realm = true
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    default_ccache_name = KEYRING:persistent:%{uid}

[realms]
     EXAMPLE.COM = {
     kdc = example.com:88
     admin_server = example.com:749
     }
     example.com = {
     kdc = example.com:88
     admin_server = example.com:749
     }

[domain_realm]
      example.com = EXAMPLE.COM
      example.com = EXAMPLE.COM

[appdefaults]
 pam = {
   debug = true
   ticket_lifetime = 36000
   renew_lifetime = 36000
   forwardable = true
   krb4_convert = false
 }
```

Furthermore, the customer's `/etc/nsswitch.conf` file:

```bash
# In order of likelihood of use to accelerate lookup.
shadow:     files
hosts:      files dns myhostname

aliases:    files
ethers:     files
gshadow:    files
# Allow initgroups to default to the setting for group.
# initgroups: files
networks:   files dns
protocols:  files
publickey:  files
rpc:        files
```

## Troubleshooting notes

Adding `debug_level = 9` to each section of the `sssd.conf` file, yields more verbose logs under `/var/log/sssd/*`

When troubleshooting the issue, the following logs are essential:

```bash
/var/log/sssd/*
/var/log/secure
```

In addition, these files are also important to check:

```bash
/etc/krb5.conf
/etc/nsswitch.conf
/etc/pam.d/password-auth
/etc/pam.d/system-auth
/etc/sssd/sssd.conf
```

Next are these command outputs to run:

```bash
getent passwd <username>
klist
kinit <username>
ssh -o PubkeyAuthentication=no -vvv -k <username>@<server_name>
```

Finally, generating an `sosreport` from the node.

## References & related articles

[Mapping local users to Kerberos principals with SSSD](https://blog.oddbit.com/post/2015-07-16-mapping-local-users-to-kerbero/)  
[sssd.conf man page](https://linux.die.net/man/5/sssd.conf)  
[sssd-krb5 man page](https://linux.die.net/man/5/sssd-krb5)


Enabling Local User Login with Active Directory Authentication Using Kerberos


## Introduction

This article explains the Name-Version-Release (NVR) format CIQ uses for LTS
packages, how to read each segment of the kernel NVR, how the FIPS-compliant
kernel naming changed in late 2025, and how to identify what is installed on a
running system.

## Userspace LTS packages

Userspace LTS packages use the standard RHEL release format with a CIQ dist
tag:

```text
<name>-<version>-<rhel-release>.el<MAJOR>_<MINOR>_ciq.<arch>.rpm
```

The `el<MAJOR>_<MINOR>_ciq` dist tag identifies the package as a CIQ LTS
build. Examples:

```text
glibc-2.28-251.el8_10_ciq.x86_64.rpm
openssl-3.0.7-27.el9_2_ciq.x86_64.rpm
```

When CIQ ships an additional patch on top of the upstream RHEL release, the
release segment is bumped (e.g., `-251.1`, `-251.2`). The dist tag stays the
same.

## Kernel NVR

Current consolidated format:

```text
kernel-<base-version>-<centos-patchset>.<rhel-patchset>.<rhel-buildid>+<ciq-patchset>.<ciq-buildid>.<dist-tag>
```

Example:

```text
kernel-5.14.0-570.60.1+2.2.el9_6_ciq.src.rpm
```

Segments:

- `5.14.0`: upstream kernel.org tag the CentOS Stream fork is based on.
- `570`: CentOS Stream patch version.
- `60`: Red Hat minor release patch version.
- `1`: Red Hat build identifier (almost always `1`).
- `+2.2`: CIQ patchset and CIQ build identifier (see below).
- `el9_6_ciq`: CIQ dist tag, identifying the major.minor base and CIQ vendor.

The `+` separator is intentional. It guarantees CIQ builds sort as newer than
the upstream RHEL build during version comparison.

### What `+X.y` means

`X` is the CIQ patchset, `y` is the CIQ build identifier, both relative to the
RHEL base in front of the `+`. The same `+2.1` can appear on different RHEL
bases and represent different content.

While a product is still rolling (RHEL base advancing), the CIQ patchset
resets each time RHEL ships a new base: CIQ rebases onto the new RHEL build,
drops anything RHEL absorbed upstream, and starts the CIQ patchset back at
`+2.1`.

```text
kernel-5.14.0-570.52.1+2.1.el9_6_ciq      ← CIQ content on top of RHEL 570.52.1
kernel-5.14.0-570.58.1+2.1.el9_6_ciq      ← rebased to 570.58.1, patchset reset
kernel-5.14.0-570.60.1+2.1.el9_6_ciq      ← rebased to 570.60.1, patchset reset
kernel-5.14.0-570.60.1+2.2.el9_6_ciq      ← second build at 570.60.1+2
```

Once a product becomes LTS (RHEL base frozen), only `+X.y` increments from
that point on:

```text
kernel-5.14.0-284.30.1+12.1.el9_2_ciq
kernel-5.14.0-284.30.1+13.1.el9_2_ciq
kernel-5.14.0-284.30.1+14.1.el9_2_ciq
```

For LTS-frozen products, `+X.y` is the field to watch when tracking which
fixes are included.

## Legacy kernel naming

Before late 2025, CIQ used product-specific suffixes (`cld_next`, `ciq_lts`,
`ciqfips`, `ciqcbr`) placed after the dist tag. Those names still exist on
systems that have not been updated. The mapping into the current naming:

```text
kernel-4.18.0-553.79.1.el8_10.cld_next.2.1   →  kernel-4.18.0-553.79.1+2.1.el8_10_ciq
kernel-4.18.0-372.32.1.el8_6.86ciq_lts.16.1  →  kernel-4.18.0-372.32.1+16.1.el8_6_ciq
kernel-5.14.0-284.30.1.el9_2.92ciq_lts.12.1  →  kernel-5.14.0-284.30.1+12.1.el9_2_ciq
kernel-3.10.0-1160.119.1.el7_9.ciqcbr.9.1    →  kernel-3.10.0-1160.119.1+9.1.el7_9_ciq
```

Version comparison works correctly across the two formats. A `dnf update` will
pull the consolidated build over a legacy build automatically.

## FIPS-compliant kernel naming

The FIPS-compliant kernel was previously published with a `ciqfips` suffix.
Starting in late 2025, the FIPS modules were merged into the consolidated
kernel build. FIPS-compliant kernels now ship under the same
`el<MAJOR>_<MINOR>_ciq` naming as the LTS kernel, with no separate `ciqfips`
identifier in the filename.

This is a packaging change. The FIPS modules are still maintained on top of
the certified kernel base, and CIQ continues to monitor upstream changes to
ensure no FIPS algorithm-impacting code is altered. Two practical effects:

1. FIPS-compliant kernels pick up CVE fixes from the upstream Rocky/RHEL
   kernel, so vulnerability scanners flag fewer issues.
2. The package filename no longer distinguishes FIPS from non-FIPS. The
   repository the package comes from is the source of truth.

If you are pulling from your product's FIPS-compliant repository, the kernel
package in that repository is the FIPS-compliant kernel for your product,
regardless of whether the filename contains `ciqfips` or just `ciq`.

For LTS 9.2 and 9.6, the FIPS and LTS kernels have been converged: the same
kernel NVR ships to both the LTS repository and the FIPS-compliant
repository. A CVE fix delivered to LTS reaches the FIPS-compliant repo at
the same NVR.

### Compliant vs. certified repositories

CIQ publishes two separate FIPS repositories per product, and they are not
interchangeable:

- **FIPS-compliant** (e.g., `fips-9.2-compliant`, `fips-9.6-compliant`):
  receives ongoing CVE fixes through the converged LTS kernel. This is the
  repository to use if your requirement is to run a FIPS-validated module
  set with current security updates. New kernel builds land here as soon as
  they are released.
- **FIPS-certified** (e.g., `fips-9.2-certified`): frozen at the formally
  certified NVR (for example,
  `kernel-5.14.0-284.30.1.el9_2.ciqfips.0.8.1`). New builds do not land in
  this repository until a new round of formal certification completes, which
  can take several months and is outside CIQ's control.

Most customer use cases do not require packages to come from the certified
repository. If you need formal certification for your deployment, work with
your CSM to confirm the current certification status for your product line.

Repository paths follow this pattern in Depot:

```text
https://depot.ciq.com/my/<repo-id>/<repo-id>-<arch>/Packages/k
```

For example:

```text
https://depot.ciq.com/my/fips-9.2-compliant/fips-9.2-compliant-x86_64/Packages/k
```

### FIPS-compliant kernel NVR transition by product

Each FIPS-compliant product transitioned from the legacy `ciqfips` suffix to
the consolidated `+X.y.el<MAJOR>_<MINOR>_ciq` form at a specific build. Any
build at or above the cutoff in that product's FIPS-compliant repository is
FIPS-compliant.

| Product | Last legacy compliant NVR | First consolidated FIPS-compliant NVR |
| --- | --- | --- |
| FIPS 8.6 | `4.18.0-553.16.1.el8_6.ciqfips.0.13.1` | `4.18.0-553.83.1+2.2.el8_6_ciq` |
| FIPS 8.10 | `4.18.0-553.16.1.el8_10.ciqfips.0.13.1` | `4.18.0-553.83.1+2.2.el8_10_ciq` |
| FIPS 9.2 | `5.14.0-284.30.1.el9_2.ciqfips.0.21.1` | `5.14.0-284.30.1+22.1.el9_2_ciq` |
| FIPS 9.6 | `5.14.0-570.52.1.el9_6_ciqfips.2.1` | `5.14.0-570.60.1+2.2.el9_6_ciq` |

Notes on the table:

- For FIPS 9.6, `5.14.0-570.56.1+2.1` and earlier `+2.1` builds on bases prior
  to `570.58.1` predate the FIPS module integration and are not
  FIPS-compliant.
- The FIPS-compliant guarantee applies to the kernel and packages built
  directly from the kernel SRPM. For other packages, refer to your product
  documentation.

## LTS kernel base reference

| LTS Product | RHEL Kernel Base |
| --- | --- |
| LTS 8.6 | `kernel-4.18.0-372.32.1.el8_6` (frozen) |
| LTS 8.8 | `kernel-4.18.0-477.27.1.el8_8` (frozen) |
| LTS 8.10 | rolling on the `4.18.0-553.x.1.el8_10` line for the lifetime of 8.10 |
| LTS 9.2 | `kernel-5.14.0-284.30.1.el9_2` (frozen) |
| LTS 9.4 | `kernel-5.14.0-427.42.1.el9_4` (frozen) |
| LTS 9.6 | `kernel-5.14.0-570.60.1.el9_6` |
| CIQ Bridge 7.9 | `kernel-3.10.0-1160.119.1.el7_9` |

For frozen LTS products, the segments before the `+` will match the base
shown above. The `+X.y` after identifies which CIQ build is installed.

## Reading the NVR on a running system

Running kernel:

```shell
uname -r
# Example: 5.14.0-284.30.1+14.1.el9_2_ciq.x86_64
```

Installed kernel package:

```shell
rpm -q kernel
# Example: kernel-5.14.0-284.30.1+14.1.el9_2_ciq.x86_64
```

All installed CIQ-built packages (covers both new and legacy naming):

```shell
rpm -qa | grep -E '_ciq($|\.|-)'
```

Available kernel versions in the configured repositories:

```shell
sudo dnf list --showduplicates kernel
```

## Notes

When comparing two systems on the same LTS product family, the segments
before the `+` should match. The `+X.y` then tells you which has more recent
CIQ content, provided both are on the same RHEL base.

If the RHEL base differs between two systems on a rolling product, `+X.y`
values are not directly comparable, since each was reset when its respective
RHEL base was bumped. Compare the full NVR.

For FIPS-compliant kernels, the repository the package was pulled from is the
authoritative answer to "is this FIPS-compliant?" The filename suffix is not
a reliable indicator on its own once you account for the legacy/consolidated
transition.

## References & related articles

[Rocky Linux Mirror List for LTS Customers](/article/rocky-linux/rl-official-mirror-list-for-lts-customers)

[Handling Vulnerabilities and CVEs](/article/rocky-linux/rl-handling-vulnerabilities-cves)


Understanding LTS Package NVRs (Name-Version-Release)


## Introduction

Optimizing NVIDIA GPU performance on Rocky Linux 8 is essential for computationally intensive tasks.

Performance inconsistencies can emerge, particularly when GPUs enter low-power states or incorrectly fall back to CPU rendering, significantly impacting workloads.

## Problem

NVIDIA GPUs have been known to intermittently drop performance to very low levels due to incorrect power states.

## Symptoms

Users experience initially acceptable GPU performance that rapidly deteriorates.

Tasks such as rotating graphical models take multiple seconds, indicating a fallback to CPU rendering or low GPU utilization.

Monitoring tools such as with `nvidia-smi` report sometimes observe GPUs stuck in performance mode P8 (low-power idle state) and show GPU utilization as minimal or zero during running of workloads.

## Resolution

### Environment Prerequisites:

* Rocky Linux 8

* NVIDIA GPU

* `root` or `sudo` privileges

* For NVIDIA GPU driver installation, [please follow this guide](https://docs.rockylinux.org/desktop/display/installing_nvidia_gpu_drivers/).

### Steps to Resolution:

The example steps below were tested using an NVIDIA 1060 GPU on Rocky Linux 8.10.

* Verify and set optimal tuned profile (for an HPC setup, the recommended profile is `hpc-compute`):

```bash
tuned-adm list
sudo tuned-adm profile hpc-compute
tuned-adm active
```

* Set GPU persistence and P-State permanently:

```bash
echo 'options nvidia NVreg_RegistryDwords="RMForcePstate=0"' | sudo tee /etc/modprobe.d/nvidia.conf
sudo dracut --force
sudo reboot
```

* GPU state checked with `nvidia-smi` before the above changes were made (observe that `P8` is listed, equivalent to `Basic HD video playback`):

```bash
|   0  NVIDIA GeForce GTX 1060 6GB    Off |
| 40%   35C    P8             10W /  120W |
```

* GPU state after the above changes with `nvidia-smi` (observe that the P-State has now changed to `P0` or `Maximum 3D performance`):

```bash
|   0  NVIDIA GeForce GTX 1060 6GB    Off |
| 40%   35C    P0             24W /  120W |
```

## Root cause

Due to incorrect GPU persistence settings, the GPU fails to be permanently set in a high power state.

## Notes

* Useful host-specific performance troubleshooting commands:

```bash
sudo dmidecode -t bios
sudo cpupower frequency-info
sudo lshw -class processor
sudo cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
tuned-adm active
tuned-adm list
```

* NVIDIA GPU-specific troubleshooting commands:

```bash
nvidia-smi
nvidia-smi -q -d PERFORMANCE
nvidia-smi -q -d POWER
nvidia-smi -q | grep -i "Persistence Mode"
nvidia-smi -q -d CLOCK
nvidia-smi --query-gpu=clocks.current.graphics,clocks.current.sm,clocks.current.memory --format=csv
```

* Ensuring persistent high-performance GPU states helps to resolve the majority of performance problems in computational and graphical workloads.

## References & related articles

[NVIDIA GPU Driver Installation on Rocky Linux](https://docs.rockylinux.org/desktop/display/installing_nvidia_gpu_drivers/)  
[NVIDIA GPU P-State Descriptions](https://docs.nvidia.com/gameworks/content/gameworkslibrary/coresdk/nvapi/group__gpupstate.html)


Maximizing NVIDIA GPU Performance on Rocky Linux 8


## Introduction

If you are subscribed to CIQ's Long Term Support (LTS) for Rocky Linux and want to move from one LTS minor version to another (for example, `9.2` LTS to `9.6` LTS, or `8.6` LTS to `8.10` LTS), you need to disable the current LTS subscription, swap the LTS release package back to the standard Rocky release, enable the new LTS subscription, and then sync packages to the target version.

**_⚠️ WARNING_** _This procedure only applies to minor-version moves within the same major release (8.x to 8.x, or 9.x to 9.x). In-place upgrades between major versions (for example, 8.x to 9.x) are **not** supported. See the multi-node migration article [here](https://kb.ciq.com/article/rocky-linux/rl-migrate-between-major-rocky-versions) for that scenario._

## Resolution

The example below migrates a system from Rocky Linux `9.2` LTS to `9.6` LTS. Substitute the version numbers (and `release` package names) for any other supported LTS pair.

1. Disable the currently enabled LTS subscription with `depot`:

    ```bash
    sudo depot disable lts-9.2
    ```

2. Swap the LTS release package back to the upstream `rocky-release` package. This removes the LTS-pinned repository definitions and restores the standard Rocky repositories:

    ```bash
    sudo dnf swap ciq-lts92-rocky-release rocky-release --allowerasing --skip-broken
    ```

3. Enable the new LTS subscription. This configures `dnf` to point to the LTS repositories for your target version:

    ```bash
    sudo depot enable lts-9.6
    ```

4. Sync the system to the target LTS packages. `distrosync` upgrades (or downgrades) packages so that everything aligns to what is available in the LTS `9.6` repositories, including the new LTS release package (`ciq-lts96-rocky-release`):

    ```bash
    sudo dnf distrosync
    ```

5. Reboot to load the new kernel and finish the transition:

    ```bash
    sudo reboot
    ```

After the reboot, verify the version and active repositories:

```bash
cat /etc/rocky-release
sudo dnf repolist
```

### Example: Migrating between `8.x` LTS versions

The same pattern applies for `8.x` LTS. To move from `8.6` LTS to `8.10` LTS:

```bash
sudo depot disable lts-8.6
sudo dnf swap ciq-lts86-rocky-release rocky-release --allowerasing --skip-broken
sudo depot enable lts-8.10
sudo dnf distrosync
sudo reboot
```

The LTS release package naming follows the pattern `ciq-lts<MAJOR><MINOR>-rocky-release` (no dot between major and minor), so `9.6` LTS uses `ciq-lts96-rocky-release` and `8.10` LTS uses `ciq-lts810-rocky-release`.

## Notes

- The `dnf swap` step replaces the LTS-pinned repository files under `/etc/yum.repos.d/` with the standard Rocky repositories. If you maintain any custom repository files, back them up first.
- Always take a backup or snapshot of the system before running the migration. Package transactions across LTS boundaries cannot be cleanly rolled back.
- If `dnf distrosync` finishes with one or two packages still unresolved, follow it with `sudo dnf upgrade` to pull in anything that was missed.
- The CIQ LTS kernel is replaced as part of the sync. If you use Secure Boot, validate the procedure on a non-production system first to confirm the new LTS kernel is signed for your environment.
- Run `sudo depot list` at any time to see which LTS subscriptions are available to your account and which one is currently enabled.

## References & related articles

[Migrate From Rocky Linux LTS to the Latest Rocky Linux Release](https://kb.ciq.com/article/rocky-linux/rl-migrate-from-lts-to-rocky)

[Migrating Between Major Rocky Linux Versions Using Multiple Nodes](https://kb.ciq.com/article/rocky-linux/rl-migrate-between-major-rocky-versions)

[Accessing Mirrors for Rocky Linux and CIQ LTS Customers](https://kb.ciq.com/article/rocky-linux/rl-official-mirror-list-for-lts-customers)

[CIQ Long Term Support](https://ciq.com/services/long-term-support/)


Migrate Between Rocky Linux LTS Versions (8.x and 9.x)


## Introduction

Frequently, businesses will want to migrate between major versions of their Linux distribution of choice in order to stay ahead of security threats, take advantage of improved hardware compatibility in a newer kernel, and other reasons for why you would want to move to a newer OS version.

This article will go over how to migrate configurations, system settings, users and more from an older system to a newer system.

**_⚠️ WARNING_** _The Rocky Linux project does not support in-place upgrades between major versions of Rocky Linux. This is also [CIQ](https://ciq.com/products/rocky-linux/)'s stance and we recommend migrating between two or more nodes or in the case of a single node, backing up the node's contents and performing a fresh install._

**_⚠️ WARNING_** _Once you complete the migration, CIQ recommends performing a human review of the migrated configurations before applying them to the new system. The reason for this is that there are thousands of changes between major Rocky Linux versions, including removed and added packages, as well as version jumps of core system components such as `kernel` and `gcc`. Incompatible packages and system configurations will happen, which is why a manual review is required._

**_⚠️ WARNING_** _The playbooks linked here have been tested on two Azure Rocky Linux VMs, as well as two baremetal servers. The testing included a migration from Rocky Linux 8 to 9 and also from 8 to 10. The playbooks are not meant to be comprehensive, and CIQ recommends expanding and modifying the playbooks to suit your environment._

## Problem

You are moving between major Rocky Linux versions. You have two or more nodes. One node is running a newer version of Rocky Linux, while the other node is running an older version of Rocky Linux, which is either experiencing hardware failure or the OS reaching end of life. You want to migrate `users`, `systemd` services, `cron jobs`, and more to the newer system, which can be a tedious and drawn-out process when done manually.

## Resolution

### Prerequisites

* Two nodes - one running an older major version of Rocky Linux and the other running a newer version.

* A separate machine configured as the Controller, responsible for orchestrating the migration between the two Rocky Linux nodes using Ansible.

### Ansible setup

Install Ansible if you have not already done so on a separate machine that is not the Source Node or Target Node. This machine will be the Controller and will orchestrate the migration between the two nodes. We recommend using `pip` to install Ansible:

```bash
# Install python3 and pip if not already available:
sudo dnf install -y python3 python3-pip
 
# Install Ansible via pip for your user:
pip3 install --user ansible

# Install system-wide if required:
sudo pip3 install ansible
```

* Ensure the required Ansible Galaxy collections are installed:

```bash
ansible-galaxy collection install -r requirements.yml
```

### Migration playbook setup

* Place all configuration files [available here](https://sftp.ciq.com/web/client/pubshares/pjKMPp5SVXM7FuXBT6b3tT?compress=false) into the same directory.

* Please go through and edit any files to better match your environment. The files that contain `<>` characters require input. An example set of playbooks has been included in the above tarball to help you.

* A key file is the `inventory.ini` file with the Source Node and Target Node IP addresses and user (ideally `root` or a user with `sudo` privileges) that you want to login with.

### Migration testing

* Copy your SSH public key to both Nodes:

```bash
ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@<SOURCE_NODE_IP>
ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@<TARGET_NODE_IP>
```

* Check that both nodes can be reached: 

```bash
ansible all -i inventory.ini -m ping
```

* You should see a similar result to the following output:

```bash
howard@ciq-rocky-linux10:~/ansible-baremetal-testing$ ansible all -i inventory.ini -m ping
source | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
target | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
```

* If the ping checks fail, ensure that `python3` is installed on all Nodes, that public and private ssh keys are available, and that the details under `inventory.ini` are correct. If going from an older version of Rocky Linux such as Rocky Linux 8.10, install a later version of `python` from the `powertools` repository:

```bash
sudo dnf config-manager --set-enabled powertools
sudo dnf install -y python39 
```

* The version of `python3` used has to be later than `python3.6`, which is installed by default on Rocky Linux 8.10 unless a newer version of `python` is installed.

* Run the `preflight-check.yaml` playbook and a total of `6` checks should report back as healthy regarding disk space availability, network connectivity, system information, and more :

```bash
ansible-playbook -i inventory.ini preflight-check.yml
```

### Migrating the data

* Start the migration between the Nodes:

```bash
ansible-playbook -i inventory.ini migrate-rocky.yml
```

* After the final phase of `Phase 5 - Cleanup SSH access` is finished, the migration will be complete.

* On the Target Node, a full report of the migration can be found under `/root/MIGRATION_COMPLETE_<UNIX_TIME>.txt`.

* Under the chosen user account's home directory of the Target Node (in this case `/root/migration_review`), you'll find migrated firewall, network, services and `ssh` configurations ported over. 

* For `/tmp/filtered_packages.txt` on the Target Node, this file has a list of the core system packages that came from the Source Node. Further exports such as security, users and system configurations can be found in `/tmp/system_export/`

### Human review

Please review the data that has been migrated over to the new node and work with your System Administrators to safely apply as many of the configurations as possible, without creating additional incompatibility errors.

## References & related articles

[Learning Ansible with Rocky Series - Rocky Linux Documentation](https://docs.rockylinux.org/books/learning_ansible/00-toc/)

[Rocky supported version upgrades - Rocky Linux Documentation](https://docs.rockylinux.org/guides/update_versions/)


Migrating Between Major Rocky Linux Versions Using Multiple Nodes


## Introduction

If you are on an LTS version of Rocky Linux and would like to transition to the latest standard Rocky Linux release, you need to revert the repositories and `release` back to defaults.

## Resolution

For example, if you were on the Rocky Linux 9.2 LTS and wanted to upgrade to Rocky Linux 9.6 (the current standard release), you can accomplish this by running these commands:

```
sudo depot disable lts-9.2
sudo dnf swap ciq-lts92-rocky-release rocky-release --allowerasing --skip-broken
sudo dnf upgrade
```

## Notes

- The `dnf swap` command will replace all of your Rocky Linux repositories in `/etc/yum/repos.d/` with the default repositories. If you use any custom repository URLs, make sure to back them up before running this command.
- The `dnf upgrade` command will upgrade your node to Rocky Linux version 9.6. Please ensure you have the necessary backups before proceeding.
- If you use `secure boot`, test this on a non-production server first, to ensure there are no issues before upgrading.

## References & related articles

[CIQ Long Term Support](https://ciq.com/services/long-term-support/)


Migrate From Rocky Linux LTS to the Latest Rocky Linux Release


## Introduction

MPI (Message Passing Interface) benchmarking is heavily utilized in the automobile industry and is a backbone of High Performance Computing. With benchmarking, the objective is to pass a message between two nodes as quickly as possible. The faster the message is passed between both nodes, the more parallel operations you will be able to perform on your selected hardware.

This article will cover how to set up MPI benchmarking on a Rocky Linux 8.10 image on two nodes in Azure.

## Prerequisites

* Two individual VMs or a Virtual Machine Scale Set (Virtual Machine Scale Set) (a VMSS is the recommended option and is what is used in this article). 

* The `size` of the VMs / VMSS needs to be able to support Infiniband and RDMA. Example `sizes` that support Infiniband and RDMA are `HBv4`, `H16r` and so on (the testing performed for this article was on a Virtual Machine Scale Set using two `HB176-24rs_v4` instances). You may need to increase your quota in your Azure subscription in order to access `sizes` such as the `HBv4`. 

* Utilize a Rocky Linux VHD to be applied to each node (for this documentation, this was tested with Rocky Linux 8.10). The image requires the `cld_next` kernel from [CIQ](https://ciq.com/products/rocky-linux/), as well as the `waagent`, `OFED`, Infiniband, and RDMA drivers. In addition, `Open MPI` needs to be installed for the MPI benchmarking to be performed.

## Azure setup

All steps are performed using the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?view=azure-cli-latest&pivots=apt).

### Step 1: create a resource group

```bash
az group create --name <RESOURCE-GROUP_NAME> --location <LOCATION> 
```

### Step 2: generate a storage account

```bash
az storage account create \
    --name <NAME> \
    --resource-group <RESOURCE-GROUP_NAME> \
    --location <LOCATION> \
    --sku Standard_LRS
```

### Step 3: provision a container

```bash
az storage container create \
    --account-name <NAME> \
    --name <NAME>
```

### Step 4: retrieve the storage account key and assign it to the `STORAGE_KEY` variable

```bash
STORAGE_KEY=$(az storage account keys list \
    --resource-group <RESOURCE-GROUP_NAME> \
    --account-name <NAME> \
    --query '[0].value' -o tsv)
```

### Step 5: display the key for verification (optional)

```bash
echo $STORAGE_KEY
```

### Step 6: upload the Rocky Linux VHD image from your local filesystem

```bash
az storage blob upload \
    --account-name <NAME> \
    --account-key $STORAGE_KEY \
    --container-name <NAME> \
    --type page \
    --file <VHD_IMAGE> \
    --name <VHD_IMAGE>
```

### Step 7: set the VHD URI to the `VHD_URI` variable

```bash
VHD_URI="https://<NAME>.blob.core.windows.net/<NAME>/<VHD_IMAGE>"
```

### Step 8: create a managed image

```bash
az image create \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <IMAGE_NAME> \
    --os-type Linux \
    --source $VHD_URI \
    --location <LOCATION>
```

### Step 9: generate an ssh key pair for accessing the nodes

```bash
# Generate an ssh key pair if you don't have one already on your local machine
ssh-keygen -t rsa -b 4096 -f ~/.ssh/<SSH_KEY> -N ""

# Check the public key for verification
cat ~/.ssh/<SSH_KEY>.pub
```

### Step 10: create a proximity placement group

```bash
az ppg create \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <PROXIMITY_PLACEMENT_GROUP_NAME> \
    --location <LOCATION> \
    --type Standard
```

### Step 11: Create a virtual machine scale set 

```bash
az vmss create \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NAME> \
    --image <IMAGE_NAME> \
    --instance-count 2 \
    --vm-sku <SIZE> \
    --admin-username azureuser \
    --ssh-key-values ~/.ssh/<SSH_KEY>.pub \
    --upgrade-policy-mode automatic \
    --lb <LOAD_BALANCER_NAME> \
    --backend-pool-name <BACKEND_POOL_NAME> \
    --location <LOCATION> \
    --public-ip-per-vm \
    --orchestration-mode Uniform \
    --ppg <PROXIMITY_PLACEMENT_GROUP_NAME>
```

### Step 12: assign public ip addresses for each node

```bash
az network public-ip create \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NODE_1_PUBLIC_IP_NAME> \
    --allocation-method Static \
    --sku Standard \
    --location <LOCATION>

az network public-ip create \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NODE_2_PUBLIC_IP_NAME> \
    --allocation-method Static \
    --sku Standard \
    --location <LOCATION>
```

### Step 13: remove any existing nat rules

```bash
az network lb inbound-nat-rule delete \
    --resource-group <RESOURCE-GROUP_NAME> \
    --lb-name <LOAD_BALANCER_NAME> \
    --name NatRule
```

### Step 14: create a nat pool

```bash
az network lb inbound-nat-pool create \
    --resource-group <RESOURCE-GROUP_NAME> \
    --lb-name <LOAD_BALANCER_NAME> \
    --name <NAT_POOL_NAME> \
    --protocol Tcp \
    --frontend-port-range-start 50000 \
    --frontend-port-range-end 50119 \
    --backend-port 22 \
    --frontend-ip-name <LOAD_BALANCER_FRONTEND_NAME>
```

### Step 15: add the nat pool to the vmss:

```bash
az vmss update \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NAME> \
    --add virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerInboundNatPools \
    '{
      "id": "/subscriptions/2247734c-d128-46cd-a462-ee14c4302d9b/resourceGroups/<RESOURCE-GROUP_NAME>/providers/Microsoft.Network/loadBalancers/<LOAD_BALANCER_NAME>/inboundNatPools/<NAT_POOL_NAME>"
    }'
```

### Step 16: check the vmss configuration:

```bash
az vmss show \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NAME> \
    --query "virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerInboundNatPools" \
    --output json
```

### Step 17: check that the nat rules were created:

```bash
az network lb inbound-nat-rule list \
    --resource-group <RESOURCE-GROUP_NAME> \
    --lb-name <LOAD_BALANCER_NAME> \
    --output table
```

### Step 18: verify each node's nics:

```bash
az vmss nic list \
    --resource-group <RESOURCE-GROUP_NAME> \
    --vmss-name <NAME> \
    --query "[].{Name:name, PrivateIP:ipConfigurations[0].privateIpAddress}" \
    --output table
```

### Step 19: get detailed information on the nics' ip configuration:

```bash
az vmss nic list \
    --resource-group <RESOURCE-GROUP_NAME> \
    --vmss-name <NAME> \
    --query "[].ipConfigurations[0]" \
    --output json
```

### Step 20: get the vmss instance information

```bash
az vmss list-instances \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NAME> \
    --output table
```

### Step 21: display the public ip of each node

```bash
az vmss list-instance-public-ips \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NAME> \
    --output table
```

### Step 22: ssh into each node

```
ssh -i ~/.ssh/<SSH_KEY> azureuser@<NODE_IP>
```

## For each node, perform the below steps:

### Step 23: generate an ssh key pair

```bash
ssh-keygen -t rsa -b 4096 -f ~/.ssh/inter_instance_key -N ""
```

### Step 24: share the public keys between both nodes

```bash
# On node 1, show the public key
cat ~/.ssh/inter_instance_key.pub

# Copy the public key

# SSH into node 2 and add node 1's public key to node 2's authorized_keys file:
echo "PASTE_NODE_1_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys

# On node 2, display the public key
cat ~/.ssh/inter_instance_key.pub

# Copy the public key

# SSH into node 1

# Add node 2's public key to node 1's authorized_keys file:
echo "PASTE_NODE_2_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys
```

### Step 25: create the hostlist.txt file for both nodes

```bash
# Get hostnames of both nodes

# On each node, create the hostlist.txt file

# First, get the hostnames of each node (run the below command on each node to identify the hostname)
hostname

# Create hostlist.txt with both hostnames
cat > ~/hostlist.txt << EOF
<NODE_1_HOSTNAME>
<NODE_2_HOSTNAME>
EOF
```

### Step 26: generate the ssh config file on both nodes

```bash
# Create ~/.ssh/config file on each instance
cat > ~/.ssh/config << 'EOF'
Host <START_OF_HOSTNAME_PATTERN>*
  IdentityFile /home/azureuser/.ssh/inter_instance_key
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
EOF
```

### Step 27: set the 600 permissions for the ssh config file on both nodes, to ensure that only the owner has read / write permissions

```bash
chmod 600 ~/.ssh/config
```

### Step 28: set 600 permissions for private ssh key

```bash
chmod 600 ~/.ssh/inter_instance_key
```

### Step 29: test ssh connectivity between the nodes

```bash
# From node 1 to node 2
ssh -i ~/.ssh/inter_instance_key <NODE_2_HOSTNAME>

# From node 2 to node 1
ssh -i ~/.ssh/inter_instance_key <NODE_1_HOSTNAME>
```

### Step 30: rdma setup for the waagent on both nodes 

```bash
sudo vi /etc/waagent.conf

# Ensure the following are enabled and uncommented
OS.EnableRDMA=y
OS.CheckRdmaDriver=y

# Save and verify the changes
grep -i rdma /etc/waagent.conf | grep -v "^#"
```

### Step 31: restart the waagent service

```bash
sudo systemctl restart waagent
sudo systemctl daemon-reload
```

### Step 32: load the ib_ipoib module

```bash
sudo modprobe ib_ipoib
```

### Step 33: configure the infiniband network

```bash
# On node 1, set an ip address (in the below example, it is 172.20.0.10)
sudo ip addr add 172.20.0.10/24 dev ib0

# Bring the link up
sudo ip link set ib0 up

# Verify the infiniband interface is up
ibdev2netdev

# On node 2, set an ip address
sudo ip addr add 172.20.0.11/24 dev ib0

# Bring the link up
sudo ip link set ib0 up

# Verify the interface is up
ibdev2netdev
```

### Step 34: test infiniband connectivity

```bash
# From node 1 to node 2 with the above example ips
ping -c 3 172.20.0.11

# From node 2 to node 1
ping -c 3 172.20.0.10
```

### Step 35: run the mpi benchmarking tests

Confirm that messages can be passed between the two nodes successfully with the `pingpong` and `allreduce` benchmarks. 

```bash
# pingpong benchmark
/usr/mpi/gcc/openmpi-4.1.7rc1/bin/mpirun -n 2 -N 1 -hostfile ~/hostlist.txt -x UCX_NET_DEVICES=mlx5_0:1 /usr/mpi/gcc/openmpi-4.1.7rc1/tests/imb/IMB-MPI1 -msglog 27:28 pingpong

# allreduce benchmark
/usr/mpi/gcc/openmpi-4.1.7rc1/bin/mpirun -n 72 -N 36 -hostfile ~/hostlist.txt -x UCX_NET_DEVICES=mlx5_0:1 /usr/mpi/gcc/openmpi-4.1.7rc1/tests/imb/IMB-MPI1 -npmin 72 allreduce
```

## Notes

### If ssh between the nodes fails

1. Verify hostnames are correct in the `hostlist.txt` file.

2. Check that public keys are properly added to `authorized_keys` file.

3. Ensure firewall rules allow internal communication.

4. Verify the ssh server service is running: `sudo systemctl status sshd`

### If Infiniband connectivity fails

1. Check if the `ib0` interface is visible: `ip addr show ib0`

2. Verify RDMA modules are loaded: `lsmod | grep rdma`

3. Check the Infiniband status: `ibstat`

4. Verify connectivity with `ibping`

### If the MPI tests fail

1. Ensure all nodes are reachable via SSH without password prompts.

2. Verify OpenMPI installation: `which mpirun`

3. Check that `UCX_NET_DEVICES` matches your hardware: `ucx_info -d`

## How to quickly cleanup a test setup and delete all resources

```bash
# Delete the entire resource group and all its resources
az group delete --name <RESOURCE-GROUP_NAME> --yes --no-wait

# Check the deletion status
az group show --name <RESOURCE-GROUP_NAME>
```

## References & related articles

[Open MPI](https://www.open-mpi.org/)


How to Perform MPI Benchmarking on Azure with Rocky Linux


## Introduction

When configuring MTU values on bonded interfaces in Rocky Linux using NetworkManager’s `nmcli`, you may find that the MTU changes aren't reflected when you inspect the interfaces (for example, via `ip a | grep mtu`). Although NetworkManager’s logs indicate that NetworkManager applies the MTU, the setting resets to the default value. This can happen when you incorrectly apply the MTU settings to the interface instead of the bond.

## Problem

The issue occurs because the MTU is only set on the slave interfaces and not on the bond interface itself. NetworkManager propagates MTU settings from the bond interface down to its slaves and any VLANs on top of it, but it doesn't propagate in the opposite direction. Without an MTU on the primary bond connection, slave interfaces inherit the default MTU.

## Symptoms

You may observe all or some of the following:

- MTU values applied via `nmcli` don't appear in the output of `ip a | grep mtu`.

- NetworkManager journal logs report the MTU change but the interfaces retain the default MTU.

- Slave interfaces such as eth1 and eth2 show the default MTU instead of the desired value.

- VLAN interfaces on top of the bond don't inherit the configured MTU.

## Resolution

### Correct configuration approach

The key to resolving this issue is to ensure the MTU is properly set in the bond interface configuration file, as MTU settings propagate from the bond interface to its slaves, but not in reverse.

#### Edit the bond interface configuration

```bash
sudo vi /etc/NetworkManager/system-connections/bond0.nmconnection
```

#### Add the MTU setting under the `[ethernet]` section

```ini
[ethernet]
mtu=9000
```

#### Remove any MTU settings from the slave interface configuration files

```bash
sudo vi /etc/NetworkManager/system-connections/eth1.nmconnection
sudo vi /etc/NetworkManager/system-connections/eth2.nmconnection
# Remove any `mtu=` lines from these files
```

### Applying the changes

There are several methods to apply these changes:

#### Reload the configuration without disruption

```bash
sudo nmcli con load /etc/NetworkManager/system-connections/bond0.nmconnection
sudo nmcli dev reapply bond0
```

This method attempts to update the device with changes to the active connection without disconnecting.

#### Restart the connection

```bash
sudo nmcli con down bond0
sudo nmcli con up bond0
```

This method is more disruptive but may be necessary if `nmcli dev reapply` doesn't work.

#### Restart NetworkManager

```bash
sudo systemctl restart NetworkManager
```

It's generally advisable to use more targeted approaches, such as reapplying the connection or bringing the interface down and back up, before opting for a full service restart.

### Verification

After applying the changes, verify the MTU settings are correctly applied:

```bash
ip a | grep mtu
```

The output should show the bond interface and all related interfaces (slaves and VLANs) with the specified MTU value.

## Notes

- MTU settings propagate from bond interfaces to their slaves, not the other way around.

- Restarting NetworkManager isn't intended to perform the same "down" and "up" actions as restarting network initscripts.

- For persistent changes, always make sure the configurations are properly set in the connection files.

- The `nmcli dev reapply` command attempts to update a device with changes made to the active connection since it was last applied.

## References

[Rocky Linux NetworkManager Gem](https://docs.rockylinux.org/10/gemstones/network/network_manager/)  
[Rocky Linux NetworkManager Documentation](https://docs.rockylinux.org/guides/network/basic_network_configuration/)


Configuring MTU Settings for Bonded Interfaces in NetworkManager


## Introduction

NFSv2 was completely removed in Rocky Linux 9. You must use Rocky Linux 8 if you desire NFSv2 support.

## Problem

NFSv2 cannot be mounted or configured on Rocky Linux 9 because support for this version of NFS has been completely removed from both the kernel and the userspace.

## Resolution

If you require NFSv2 support, the recommended solution is to migrate to Rocky Linux 8.10. This version includes support for NFS versions 2, 3, 4.0, 4.1, and 4.2. Rocky Linux 8.10 is fully supported and will remain so until May 31, 2029. CIQ offers further extended support for Rocky Linux 8.10 through its [Long-Term Support (LTS) program](https://ciq.com/services/long-term-support/).

While you can compile your own kernel for Rocky Linux 9 that supports NFSv2, CIQ doesn't recommend this.

## Notes

NFSv2 was removed because it’s an outdated, insecure protocol limited to 32-bit file sizes, lacking modern features like ACLs, and strong authentication. Its continued support added unnecessary complexity and security risk, so both user-space tools and kernel support were dropped in favor of NFSv3 and v4. Users are encouraged to migrate to newer NFS versions.

## References

[Rocky Linux NFS Guide](https://docs.rockylinux.org/labs/networking/lab5-nfs/?h=nfs)  
[Rocky Linux Support Timelines](https://wiki.rockylinux.org/rocky/version/)  
[RLC Pro LTS from CIQ](https://ciq.com/services/long-term-support/)


NFSv2 Support Removed in Rocky Linux 9


## Introduction

NFSv3 file locking requires additional configuration compared to NFSv4. This article covers the proper setup of NFSv3 file locking on Rocky Linux.

## Problem

NFSv3 file locking operations are failing or timing out. Applications report issues obtaining file locks, and `flock` commands timeout on NFSv3 mounts.

## Resolution

### Verify rpc-statd service status

NFSv3 requires the `rpc-statd` service to be running on both the server and all clients for file locking to work properly.

Check the status of rpc-statd:

```bash
systemctl status rpc-statd
```

If the service isn't running, enable, and start it:

```bash
systemctl enable rpc-statd
systemctl start rpc-statd
```

### Configure fixed ports for NFS services

By default, NFSv3 RPC services use random ports, which can cause firewall issues. To configure fixed port numbers, edit the `/etc/nfs.conf` file and add or modify the following sections:

```ini
[lockd]
port=5555

[statd]
port=6666
```

### Configure firewall rules

If `firewalld` is enabled, whitelist the NFS services and the fixed ports:

```bash
firewall-cmd --permanent --add-service=rpc-bind
firewall-cmd --permanent --add-service=nfs
firewall-cmd --permanent --add-port={5555/tcp,5555/udp,6666/tcp,6666/udp}
firewall-cmd --reload
```

### Restart services

Restart the necessary services to apply the configuration changes:

```bash
systemctl restart rpc-statd nfs-server
```

### Verify configuration

To test file locking is working correctly, perform a simple test showing that one client waits for another to release the lock.

#### Test from Client 1

```bash
flock -x /path/to/nfs/mount/file.txt -c 'echo "Client 1 holding the lock"; sleep 30'
```

#### Test from Client 2 (run this while Client 1 is still holding the lock)

```bash
flock -x /path/to/nfs/mount/file.txt -c 'echo "Client 2 holding the lock"; sleep 10'
```

#### Expected behavior

**Client 1** should immediately display "Client 1 holding the lock" and hold the lock for 30 seconds.
**Client 2** should wait until Client 1's 30-second timer finishes before displaying "Client 2 holding the lock".

#### If file locking is working correctly

Client 2 should wait approximately 30 seconds before executing.

#### If file locking isn't working

Client 2 executes immediately without waiting for Client 1 to release the lock. Both clients may display their messages simultaneously.

## Root cause

NFSv3 uses the Network Lock Manager (NLM) sidecar protocol for file locking, which requires the `rpc-statd` service. Unlike NFSv4, which has file locking built into the protocol, NFSv3 relies on separate RPC services that must be properly configured and accessible through firewall rules.

## Notes

- If using a different NFS server implementation (like Dell OneFS), verify that the server also supports and has NLM properly configured.

- The `flock` testing only tests accurately between 2 NFS clients and not necessarily between a client and the server itself.

- To avoid potential data corruption, always access NFSv3 shares via NFS rather than locally on the server, since local and remote file locks don't interact.

## References & related articles

[Rocky Linux NFS Documentation](https://docs.rockylinux.org/guides/file_sharing/nfsserver/)  
[Using flock](https://linuxhandbook.com/flock-command/)


Configuring NFSv3 File Locking on Rocky Linux


## Introduction

The Nagios Remote Plugin Executor (NRPE) plugin is fundamental when combined with Nagios for adequate monitoring and provides you with system metrics including CPU, memory, disk usage and more from your Linux and Windows fleet.

This article will detail how to setup a basic Nagios Core server-client environment on Rocky Linux 9.5, how to install the NRPE plugin, and provide example NRPE commands with which to check the status of your Nagios clients.

## Prerequisites

* Two baremetal or virtual nodes running Rocky Linux 9.5, with a minimum of 1 vCPU, 512MB RAM, and 500MB of disk space. The recommendation for Rocky Linux is 4 vCPU cores, 4096MB RAM and 10GB of disk space.

* Access to either the `root` account or a user with `sudo` privileges.

## Nagios Core server configuration

### Server initial setup

* To install the latest version of Nagios Core Server, it is recommended to build directly from source, as opposed to downloading the packages from the EPEL repository.

* Upgrade all packages:

```bash
dnf upgrade -y
```

* Reboot the server:

```bash
reboot
```

* Install the required packages for Nagios:

```bash
dnf install -y gcc gd gd-devel glibc glibc-common httpd perl php openssl-devel wget
```

* Install the `Development Tools` group to aid with the source compilation process:

```bash
dnf groupinstall -y "Development Tools"
```

* Switch to the `tmp` directory:

```bash
cd /tmp
```

* Pull down the latest Nagios Core source code:

```bash
wget --output-document="nagioscore.tar.gz" $(wget -q -O - https://api.github.com/repos/NagiosEnterprises/nagioscore/releases/latest  | grep '"browser_download_url":' | grep -o 'https://[^"]*')
```

* Extract the tarball:

```bash
tar xzf nagioscore.tar.gz
```

* Change into the `nagios` directory (in this example, the version is `4.5.9`):

```bash
cd /tmp/nagios-4.5.9/
```

* Run `configure` to make sure all required dependencies are available:

```bash
./configure
```

* Run `make all` to compile the source code:

```bash
make all
```

* Generate the `nagios` user and group:

```bash
make install-groups-users
```

* Add the `apache` user that was generated from the previous `dnf` package installs into the `nagios` group:

```bash
usermod -a -G nagios apache
```

* Install the required binaries, CGIs and HTML files:

```bash
make install
```

* Install the `nagios` daemon:

```bash
make install-daemoninit
```

* Download the latest plugins pack:

```bash
wget --output-document="nagios-plugins.tar.gz" $(wget -q -O - https://api.github.com/repos/nagios-plugins/nagios-plugins/releases/latest  | grep '"browser_download_url":' | grep -o 'https://[^"]*')
```

* Unpack the tarball:

```bash
tar -xf nagios-plugins.tar.gz
```

* Change into the plugins directory, in this case it is version `2.4.12`:

```bash
cd nagios-plugins-2.4.12/
```

* Check for dependencies:

```bash
./configure
```

* Compile from source:

```bash
make
```

* Install the plugins to the appropriate directories:

```bash
make install
```

* Confirm the plugins were installed correctly. The number of plugins should be between `60~70`:

```bash
ls -l /usr/local/nagios/libexec/ | wc -l
```

* Set the `httpd` `systemd` service to restart on the next reboot:

```bash
systemctl enable httpd
```

* Similarly, configure the `nagios` service to start on a reboot:

```bash
systemctl enable nagios
```

* Setup `Command Mode`:

```bash
make install-commandmode
```

* Install the required configuration files:

```bash
make install-config
```

* Install the Apache web server files:

```bash
make install-webconf
```

* Configure `firewalld` to allow traffic on TCP port `80`:

```bash
firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --reload
```

* Create the `nagiosadmin` account and set a password:

```bash
htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin
```

* Start the `httpd` service:

```bash
systemctl start httpd
```

* Start the `nagios` service:

```bash
systemctl start nagios
```

### NRPE plugin installation

* Check the [NRPE plugin releases](https://github.com/NagiosEnterprises/nrpe/releases/) and download the latest version. As of writing, this is `4.1.3`:

```bash
wget https://github.com/NagiosEnterprises/nrpe/releases/download/nrpe-4.1.3/nrpe-4.1.3.tar.gz
```

* Extract the tarball:

```bash
tar -xf nrpe-4.1.3.tar.gz
```

* Change into the `nagios` directory (in this case version `4.1.3`):

```bash
cd nrpe-4.1.3/
```

* Check for dependency issues:

```bash
./configure
```

* Compile from source:

```bash
make all
```

* Install the plugin daemon and sample config files:

```bash
make install-daemon
make install-init
make install-plugin
```

## Nagios Core client configuration

### Client initial setup

* Make sure all packages are up to date:

```bash
dnf upgrade -y
```

* Reboot the client machine:

```bash
reboot
```

* Install required packages:

```bash
dnf install -y automake autoconf gcc gd gd-devel gettext glibc glibc-common make net-snmp net-snmp-utils openssl-devel tar wget
```

* Generate the `nagios` user and assign a password:

```bash
useradd nagios
passwd nagios
```

* Change into the `tmp` directory:

```bash
cd /tmp
```

* Download the plugins pack:

```bash
wget --output-document="nagios-plugins.tar.gz" $(wget -q -O - https://api.github.com/repos/nagios-plugins/nagios-plugins/releases/latest  | grep '"browser_download_url":' | grep -o 'https://[^"]*')
```

* Extract the tarball:

```bash
tar -xf nagios-plugins.tar.gz
```

* Change into the plugins directory (in this case the version is `2.4.12`):

```bash
cd nagios-plugins-2.4.12/
```

* Check for dependency issues:

```bash
./configure
```

* Compile from source:

```bash
make
```

* Install the required packages:

```bash
make install
```

* Ensure the plugins were installed correctly (there are usually `60~70` plugins available):

```bash
ls -l /usr/local/nagios/libexec/ | wc -l
```

* Configure directory permissions for the `nagios` user:

```bash
chown nagios.nagios /usr/local/nagios
chown -R nagios.nagios /usr/local/nagios/libexec
```

### NRPE installation

* Pull down the latest version of the NRPE plugin (as mentioned above, the current version is `4.1.3`):

```bash
wget https://github.com/NagiosEnterprises/nrpe/releases/download/nrpe-4.1.3/nrpe-4.1.3.tar.gz
```

* Extract the tarball:

```bash
tar -xf ./nrpe-4.1.3.tar.gz
```

* Change into the `nagios` directory:

```bash
cd nrpe-4.1.3/
```

* Check for any dependency issues:

```bash
./configure
```

* Compile the NRPE daemon and client:

```bash
make all
```

* Install the plugin daemon and sample config files:

```bash
make install-config
make install-daemon
make install-plugin
```

* Create a `systemd` service for the NRPE daemon:

```bash
make install-init
```

* Enable the NRPE daemon `systemd` service to start on next boot:

```bash
systemctl enable nrpe
```

* Configure the NRPE plugin configuration file at `/usr/local/nagios/etc/nrpe.cfg` and add the address of your Nagios server. In the example illustrated below, the Nagios server address is `10.0.0.11`:

```bash
sed -i 's/allowed_hosts=127.0.0.1,::1/allowed_hosts=127.0.0.1,10.0.0.11/' /usr/local/nagios/etc/nrpe.cfg
```

* Open TCP port `5666` in `firewalld`:

```bash
firewall-cmd --zone=public --add-port=5666/tcp --permanent
firewall-cmd --reload
```

* Start the `nrpe` service:

```bash
systemctl start nrpe
```

* Verify that NRPE was installed successfully:

```bash
/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1
```

## Commands to check metrics on a Nagios client from the server

* In the following examples, the Nagios client has an IP of `10.0.0.12`:

* Initial check that the Nagios client can be reached:

```bash
/usr/local/nagios/libexec/check_nrpe -H 10.0.0.12
NRPE v4.1.3
```

* Check the amount of logged-in users:

```bash
/usr/local/nagios/libexec/check_nrpe -H 10.0.0.12 -c check_users
USERS OK - 1 users currently logged in |users=1;5;10;0
```

* Check the load on the client system:

```bash
/usr/local/nagios/libexec/check_nrpe -H 10.0.0.12 -c check_load
OK - load average per CPU: 0.00, 0.01, 0.01|load1=0.000;0.150;0.300;0; load5=0.007;0.100;0.250;0; load15=0.010;0.050;0.200;0;
```

## Notes

* The above configurations for server and client work on both FIPS and non-FIPS systems.

* `nagios` logs are located under `/usr/local/nagios/var/nagios.log`.

* Further logs and outputs can be found by running `journalctl -exu nrpe` and `netstat -at | grep nrpe`.

* Make sure the latest plugins pack is installed. If not, you will observe `NRPE: Unable to read output` in your terminal, when running commands such as `/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1 -c check_load`.

## References & related articles

* [Nagios Installation Guide for Rocky Linux](https://support.nagios.com/kb/article/nagios-core-installing-nagios-core-from-source-96.html#RHEL)

* [NRPE Plugin](https://exchange.nagios.org/directory/Addons/Monitoring-Agents/NRPE--2D-Nagios-Remote-Plugin-Executor/details)

* [NRPE Releases](https://github.com/NagiosEnterprises/nrpe/releases)


Installing the NRPE Plugin for Nagios Core on Rocky Linux


## Introduction

Customers migrating to Rocky Linux or using CIQ LTS support need reliable and accurate access to the correct software repositories.

This document provides the official and complete list of public mirrors and repositories that CIQ and Rocky Linux users should use.

It ensures customers can correctly configure their systems for `migrate2rocky`, CIQ LTS support, and package management across all supported versions.

## Problem

When customers migrate from Red Hat Enterprise Linux, CentOS, or other distributions, they may not know which mirror or repository to use. They risk accessing outdated or unsupported packages, or broken links during migration.

CIQ LTS customers specifically need to access their distribution's matching LTS repository version to ensure stability.

## Symptoms

Systems fail to update correctly after using the [migrate2rocky](https://docs.rockylinux.org/guides/migrate2rocky/) script.

`dnf` or `yum` errors referencing missing packages, 404s, or unexpected versions.

CIQ LTS customers pulling down incorrect Rocky Linux versions not aligned with their support agreement.

## Resolution

### Main Rocky Linux repositories

[`download.rockylinux.org`](https://download.rockylinux.org) and [`dl.rockylinux.org`](https://dl.rockylinux.org) are the official repositories for Rocky Linux.

### Rocky Linux public mirrors

For official mirrors, there is a list at [https://mirrors.rockylinux.org](https://mirrors.rockylinux.org)

Choose the Rocky Linux version and architecture you desire and then select the mirror that is geographically closest to you.

### CIQ LTS Customers

For [CIQ LTS](https://ciq.com/services/long-term-support/) Customers, the format of the URL is `https://depot.ciq.com/files/<repository_name>/<repository_name>-<architecture>`

An example for the Rocky Linux 9.6 LTS `AppStream` would be `https://depot.ciq.com/files/lts-9.6/rocky-appstream-9.6.x86_64`

[Once CIQ Depot is installed](https://portal.ciq.com/docs/getting-started) and you have logged in, running `sudo depot list` will show the available repositories you can subscribe to.

## Root cause

Customers can unintentionally utilise mirrors that are either not in their geographical region, are not updated enough.  

This results in mismatched versions, unsupported configurations or broken links when pulling packages.

## Notes

Ensure to enable the correct firewall ports to allow access to the mirror of your choice. The ports will be TCP ports `80` for HTTP and `443` for HTTPS.

## References & related articles

[CIQ Depot Documentation](https://portal.ciq.com/docs/depot-guide)

[`dl.rockylinux.org`](https://dl.rockylinux.org)

[`download.rockylinux.org`](https://download.rockylinux.org)

[Integrating CIQ Repositories with Red Hat Satellite Guide](https://portal.ciq.com/docs/satellite-guide)

[Mirroring CIQ Repositories Using `reposync` Guide](https://portal.ciq.com/docs/reposync-guide)

[Mirroring CIQ Repositories Using `rsync` Guide](https://portal.ciq.com/docs/rsync-guide)

[Rocky Linux Mirror List](https://mirrors.rockylinux.org)

[Rocky Linux Repositories](https://wiki.rockylinux.org/rocky/repo/#notes-on-difference-between-x-and-xy-in-mirrors)


Accessing Mirrors for Rocky Linux and CIQ LTS Customers


## Introduction

A `mutex` (meaning `mutual exclusion`) prevents multiple threads from accessing the same resource simultaneously. When [CIQ](https://ciq.com/products/rocky-linux/) was running a `futex` test (a form of testing that checks the Linux kernel `futex` system call API), we observed `mutex` slowdown when running the `futex` testing on Rocky Linux 8.9 compared with CentOS 7.9.

We investigated this performance discrepancy further and this article will reveal why there was a performance differential between the two operating systems.

## Resolution

The performance differences between both operating systems were due to the BIOS settings on the HPE Blade machines that were used for testing. The CPU governor settings were set to `balanced power saving` mode.

Upon configuring the CPU governor to `static high performance` mode in the BIOS, we saw a dramatic increase in performance for Rocky Linux 8.9.

Running the `futex` tests again on both Rocky Linux 8.9 and CentOS 7.9 produced the performance results as expected.

We also advise setting the `tuned-adm` profile to `hpc-compute` to ensure maximum performance from Rocky Linux.

## Conclusion

When comparing performance between a Rocky Linux X.X version and another operating system, CIQ advises checking that the BIOS settings have the CPU governor configured for maximum performance.

In addition, we advise setting the `tuned-adm` profile to `hpc-compute` to ensure maximum performance from Rocky Linux.

## References

- [Futex Testing from the Linux Kernel Docs Team](https://www.kernel.org/doc/readme/tools-testing-selftests-futex-README)

- [Generic Mutex Subsystem from the Linux Kernel Docs Team](https://docs.kernel.org/locking/mutex-design.html)

- [Set the TuneD Profile via a Config File - CIQ Knowledge Base](https://kb.ciq.com/article/rocky-linux/rl-tuned-profile-set-via-config-file)


Performance Differences Between CentOS 7.9 and Rocky Linux 8.9 - A Root Cause Analysis and Recommendations


## Introduction

`cgroups v2` is an excellent evolution over `cgroups v1` and provides additional functionality such as enhanced resource allocation management, resource isolation, Pressure Stall Information, and more.

When enabling `cgroups v2` on Rocky Linux 8 however (which uses `cgroups v1` by default) for use with `podman`, there are areas of the system including your `containers.conf` and `pam` configurations, that need to be checked and updated if necessary.

## Problem

You run `sudo grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="systemd.unified_cgroup_hierarchy=1"` and then reboot your Rocky Linux 8 system to enable `cgroups v2`. However, upon running a container with `podman run`, you observe this `dbus` error:

```
dbus: couldn't determine address of session bus  Failed to create bus connection: No such file or directory
```

## Symptoms

The following commands should provide output for your user. If not, that usually means the `pam` configuration is too strict on the system and is not allowing the user's `dbus` session to run:

```bash
systemctl status user@$(id -u).service
systemctl status user-$(id -u).slice
echo $XDG_RUNTIME_DIR
```

## Resolution

Your system-wide `containers.conf` configuration at `/etc/containers/containers.conf` or at the user-level at `$HOME/.config/containers/containers.conf` should contain the below line under `[engine]`:

```text
cgroup_manager = "cgroupfs"
```

In addition, check your `pam` configuration. A configuration that is too strict, can cause the user's `dbus` session to not run and thus the `couldn't determine address of session bus` errors appear when executing `podman`. Adding the line `session optional pam_systemd.so` in `/etc/pam.d/sshd` is not enough, the same line also needs to be set in `/etc/pam.d/password-auth` and `/etc/pam.d/system-auth` as well, to ensure the user's `dbus` session activates and runs without interruption. 

## Root Cause

`cgroup_manager = "cgroupfs"` not being set in either the system-level `/etc/containers/containers.conf` file or the user-level `$HOME/.config/containers/containers.conf` configuration.

`session optional pam_systemd.so` only being set in `/etc/pam.d/sshd` and not in `/etc/pam.d/password-auth` and `/etc/pam.d/system-auth` as well.

## References & related articles

[About cgroup v2 - Kubernetes Documentation](https://kubernetes.io/docs/concepts/architecture/cgroups/)
[Lab 4: Advanced System and process monitoring - Rocky Linux Documentation](https://docs.rockylinux.org/10/labs/systems_administration_II/lab4-advanced_system_process_monitoring/)
[Podman Documentation](https://docs.podman.io/en/stable/markdown/podman.1.html)
[What does pam_systemd.so do?](https://unix.stackexchange.com/questions/626665/what-does-pam-systemd-so-do)


Avoiding dbus Errors with cgroups v2 and podman on Rocky Linux 8


## Introduction

Inside a rootless Podman container on Rocky Linux, `dnf update` can fail with `cpio: chown failed` errors. The cause is how the container maps user IDs to the host.

## Problem

Some packages fail to install during `dnf update` in a rootless container such as `rockylinux:8.10`.

## Symptoms

```text
error: unpacking of archive failed on file /etc/tcsd.conf: cpio: chown failed - No such file or directory
error: unpacking of archive failed on file /var/lib/unbound: cpio: chown failed - Directory not empty
error: trousers-0.3.15-2.el8.x86_64: install failed
```

The error may also read `Invalid argument` or `Operation not permitted`.

## Root Cause

A rootless container runs in a user namespace, mapping its user and group IDs to a subordinate ID range on the host (`/etc/subuid`, `/etc/subgid`). When `rpm` sets a file's owner to an ID outside that range, the chown fails. The range must cover every ID the packages use; 65536 is the standard allocation. It breaks when the range is missing, too small, or differs between hosts that share the user's storage.

## Resolution

### Set a full ID range

Check the ranges:

```bash
grep '^user:' /etc/subuid /etc/subgid
```

Each line should grant at least 65536 IDs (for example `user:100000:65536`). If it is missing or smaller, set it as root and apply it:

```bash
sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 user
podman system migrate
```

`podman system migrate` applies the new mapping. Retry `dnf update` in a fresh container. The `newuidmap` and `newgidmap` helpers (`shadow-utils`) must be installed.

### Keep storage on local disk

Podman does not support storage on NFS. If `graphroot` is on NFS, move it to a local `XFS` or `ext4` path in `~/.config/containers/storage.conf`, then re-pull images:

```ini
[storage]
driver = "overlay"
graphroot = "/home/user/.local/share/containers/storage"
```

### Single-UID hosts: ignore chown errors

If a full range is not possible (for example a single mapped UID), squash image IDs to your own UID:

```ini
[storage.options.overlay]
ignore_chown_errors = "true"
```

This removes UID separation in the image, so use it only as a fallback.

## Notes

Inspect the live mapping: `podman unshare cat /proc/self/uid_map`.

Sharing one user's storage across hosts is fragile even with matching ranges. Prefer local storage per host.

## References & related articles

[Podman rootless mode tutorial](https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md)

[Rocky Linux: Rootless Podman (advanced)](https://docs.rockylinux.org/10/guides/containers/rootless_podman_advanced/)

[containers-storage.conf(5)](https://github.com/containers/storage/blob/main/docs/containers-storage.conf.5.md)

[Avoiding dbus Errors With cgroups v2 and podman on Rocky Linux 8](https://kb.ciq.com/article/rocky-linux/rl-podman-rl8-cgroupsv2-pam)


Rootless Podman Fails to dnf Update With cpio chown Errors


## Introduction

In some environments, such as HPC clusters, rootless Podman containers should reach the local network but not the internet. You get this by putting them on an `internal` Podman network.

## Problem

You need rootless containers limited to a controlled network with no internet access, while still allowing the containers to reach each other.

## Resolution

### Create an internal network (`netavark` backend)

On Rocky Linux 9 (and Rocky 8.x using `netavark`), an internal network has no route to the outside, so containers on it cannot reach the internet, but containers on the same network can still reach each other. Create one:

```bash
podman network create --internal no-inet-net
```

Run containers on it with `--network`:

```bash
podman run --rm -it --network no-inet-net rockylinux:9 bash
```

Tested on Rocky Linux 9.8 (podman 5.8.2): a container on the internal network has no internet, while one on the default network still does.

```text
# on the internal network
ping -c2 8.8.8.8
ping: sendto: Network unreachable
```

To control the address range, set the subnet:

```bash
podman network create --internal --subnet 10.0.0.0/24 no-inet-net
```

### Older setups using the CNI backend

`netavark` is the default since Podman 4.0 (fresh Rocky 9); a system upgraded from Rocky 8 can still use the older `cni` backend. Check it:

```bash
podman info --format '{{.Host.NetworkBackend}}'
```

If it reports `cni`, define the network as a CNI bridge. Create `~/.config/cni/net.d/no-inet-net.conflist`:

```json
{
  "cniVersion": "0.4.0",
  "name": "no-inet-net",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni-podman1",
      "isGateway": false,
      "ipMasq": false,
      "ipam": {
        "type": "host-local",
        "ranges": [[{ "subnet": "10.0.0.0/24" }]]
      }
    }
  ]
}
```

Here `ipMasq: false` disables the source NAT that containers use to reach the internet, and `isGateway: false` leaves the bridge with no gateway or default route off-host, together approximating `--internal`. Run containers with `--network no-inet-net`. CNI was removed in Podman 5.0, so this applies only to older hosts.

## Notes

The network is per user: each rootless user must create it. The `default_network` setting in `containers.conf` does not help, because it applies only to containers run as root, not to rootless ones. Since a user can still pick another network with `--network`, enforce the isolation at the host firewall if it must be guaranteed. For no network access at all, use `--network none`.

## References & related articles

[podman-network-create](https://docs.podman.io/en/latest/markdown/podman-network-create.1.html)

[Rocky Linux: Rootless Podman (advanced)](https://docs.rockylinux.org/10/guides/containers/rootless_podman_advanced/)

[Rootless Podman Fails to dnf Update With cpio chown Errors](https://kb.ciq.com/article/rocky-linux/rl-podman-rootless-chown-dnf)


Restricting Rootless Podman Containers to a Local Network Without Internet Access


## Introduction

Using PXE to bootstrap nodes with an image is commonplace in many business environments and has made both deploying and managing an image across a fleet of nodes at scale trivial.

This article will go through the setup process of deploying a PXE server on Rocky Linux 9.7 and then bootstrap a fresh node with a Rocky Linux 9.7 image. The article will focus on a UEFI boot use case, as opposed to a legacy BIOS boot.

## Prerequisites

* A node with Rocky Linux 9.7 installed on it. This will be the PXE server.

* A node that has PXE support available. This will be the PXE client.

* A Rocky Linux 9.7 DVD ISO.

## Instructions

### PXE Server Setup

To set up the PXE Server please follow the steps below:

* Update all packages:

```bash
sudo dnf update -y
```

* Install `dnsmasq` to provide a DNS and DHCP server support:

```bash
sudo dnf install -y dnsmasq
```

* Set up your `dnsmasq.conf` configuration with the appropriate subnet, netmask, broadcast-address, and more:

```bash
cat << "EOF" | sudo tee /etc/dnsmasq.conf
interface=<your_ethernet_interface>,lo
#bind-interfaces
domain=<name_here-anything_is_okay>
# DHCP range-leases
dhcp-range=<your_ethernet_interface>,<start_of_dhcp_range>,<end_of_dhcp_range>,255.255.240.0,1h
# PXE
dhcp-boot=pxelinux.0,PXEserver,<your_rocky_linux_node_address>
# Gateway
dhcp-option=3,<your_gateway_address>
# DNS
dhcp-option=6,<your_dns_server_address>
server=<your_rocky_linux_node_address>
# Broadcast Address
dhcp-option=28,<your_broadcast_address>
# NTP Server
dhcp-option=42,0.0.0.0

pxe-prompt="Press F8 for menu.", 60
pxe-service=x86PC, "Install Rocky Linux 9.7 from your PXE server <your_rocky_linux_node_address>", pxelinux
enable-tftp
tftp-root=/var/lib/tftpboot
EOF
```

Example:
*Please supplement the addresses with those from your own environment*

```bash
cat << "EOF" | sudo tee /etc/dnsmasq.conf
interface=enp8s0,lo
#bind-interfaces
domain=test
# DHCP range-leases
dhcp-range=enp8s0,10.25.96.4,10.25.96.5,255.255.240.0,1h
# PXE
dhcp-boot=pxelinux.0,pxeserver,10.25.96.3
# Gateway
dhcp-option=3,10.25.96.1
# DNS
dhcp-option=6,10.25.96.1
server=10.25.96.3
# Broadcast Address
dhcp-option=28,10.25.96.255
# NTP Server
dhcp-option=42,0.0.0.0

pxe-prompt="Press F8 for menu.", 60
pxe-service=x86PC, "Install Rocky 9.7 from network server 10.25.96.3", pxelinux
enable-tftp
tftp-root=/var/lib/tftpboot
EOF
```

*The PXE server has been assigned the address of `10.25.96.3` in the above configuration.*

* Install the `syslinux` package:

```bash
sudo dnf install -y syslinux
```

* Install the `tftp-server` package:

```bash
sudo dnf install -y tftp-server
```

* Copy the contents of the `syslinux` directory to `/var/lib/tftpboot`:

```bash
sudo cp -r /usr/share/syslinux/* /var/lib/tftpboot
```

* Create a directory called `pxelinux.cfg`:

```bash
sudo mkdir /var/lib/tftpboot/pxelinux.cfg
```

* Generate the PXE BOOT menu:

```bash
cat << "EOF" | sudo tee /var/lib/tftpboot/pxelinux.cfg/default
default menu.c32
prompt 0
timeout 300
ONTIMEOUT local

menu title # Rocky Linux 9.7 Boot Menu #

label 1
menu label ^1) Install Rocky Linux 9.7 x64 using a Remote Repository
  kernel rocky97/vmlinuz
  append initrd=rocky97/initrd.img inst.zram=1 inst.repo=https://dl.rockylinux.org/pub/rocky/9.7/BaseOS/x86_64/os/
EOF
```

**NOTE**
The PXE BOOT menu can be further expanded to include `kickstart` file configurations and other customizations. An example entry using a `kickstart` file is below:

```bash
label 2
menu label ^2) Install Rocky Linux 9.7 x64 using a Remote Repository
  kernel rocky97/vmlinuz
  append initrd=rocky97/initrd.img inst.debug ip=dhcp inst.ks=nfs:<your_rocky_linux_node_ip>:</path/to/your/kickstarter_cfg_file> inst.repo=nfs:<your_rocky_linux_node_ip>:</path/to/your/iso> inst.zram=1
```

* Create a directory called `rocky97`:

```bash
sudo mkdir /var/lib/tftpboot/rocky97
```

* Mount the Rocky Linux 9.7 DVD ISO image:

```bash
sudo mount -o loop </path/to/iso> </iso/mount_point>
```

* For this example we mounted the Rocky Linux 9.7 ISO at `/mnt`:

```bash
sudo mount -o loop ~/isos/Rocky-9.7-x86_64-dvd.iso /mnt
```

* Copy the required `initrd.img` and `vmlinuz` files to `/var/lib/tftpboot/rocky97`:

```bash
sudo cp /mnt/images/pxeboot/vmlinuz /var/lib/tftpboot/rocky97
sudo cp /mnt/images/pxeboot/initrd.img /var/lib/tftpboot/rocky97
```

* Start and enable the `dnsmasq` and `tftp` services at boot:

```bash
sudo systemctl enable --now dnsmasq
sudo systemctl enable --now tftp
```

* Allow `dhcp`, `dns`, `proxydhcp` and `tftp` traffic through your firewall:

```bash
sudo firewall-cmd --add-service=dhcp --permanent
sudo firewall-cmd --add-service=dns --permanent
sudo firewall-cmd --add-port=69/udp --permanent
sudo firewall-cmd --add-port=4011/udp --permanent
sudo firewall-cmd --reload
```

### PXE Client Setup

To set up the PXE Client please follow the steps below to enable the available PXE option from the client machine's BIOS settings. In this example, enable the following:

* Navigate to `Advanced` --> `Onboard Devices Configuration` --> `Realtek LAN Controller` and set this option to `Enabled`.

* Navigate to `Advanced` --> `Onboard Devices Configuration` --> `Realtek LAN Controller` --> `Realtek PXE Option ROM` and set this option to `Enabled`.

* Navigate to `Advanced` --> `Network Stack Configuration` --> `Network Stack` and also set this option as `Enabled`.

* Navigate to `Advanced` --> `Network Stack Configuration` --> `Ipv4 PXE Support` and set this option to `Enabled`.

* Restart your PXE client machine and it should connect to your PXE server.

* Press `F8` at the prompt:

![The prompt to press F8 shows that the PXE client machine has connected to the PXE server successfully. Press F8 to continue.](/images/articles/pxe_server_boot_1.webp)

* Press the `Enter` key at the next screen and proceed to the Rocky Linux 9.7 Boot Menu. Select `option 1`:

![The Rocky Linux 9.7 PXE Boot Menu appears. One option called "Install Rocky Linux 9.7 x64 using a Remote Repository" is present. The user is prompted to select that option.](/images/articles/pxe_server_boot_2.webp)

Your PXE client will now boot up and download the `initrd.img` and `vmlinuz` files, as well as the files from the remote repository. The Anaconda installer will then appear and you can continue the installation as normal. Of course, if you submit a `kickstart` configuration file, the installation part of the process can be skipped.

## Conclusion

This guide was built in the hope of helping bring up a simple PXE server. It touched on configurations for `dnsmasq`, `tftp`, setting up the PXE server boot menu, and more. PXE is an amazing technology and has truly revolutionized large scale deployments.


How to Setup a PXE Server on Rocky Linux 9.x


## Introduction

When provisioning Rocky Linux via PXE using NFS, you can't host both the installer source and your kickstart file on the same NFS share. This article explains how to work around this limitation by creating two separate exports, one for the kickstart file and one for the installation media.

## Problem

Attempting to point both `inst.ks` and `inst.repo` to the same NFS share results in a conflict. The installer fails to locate one of the resources, causing the automated installation to fail. For example, hosting your kickstart in `nfs:10.1.0.15:/mnt/repo/ks.cfg` and your source at `nfs:10.1.0.15:/mnt/repo/` doesn't work. As a result, you may encounter *access denied* or *failed to mount* errors.

## Resolution

### Prerequisites

`sudo` privileges are required.

### NFS configuration

On your NFS server, create two directories:

- `/mnt/ks`: for the kickstart file  
- `/mnt/repo`: for the Rocky Linux ISO or repository

Configure exports in `/etc/exports`:

```bash
/mnt/ks    10.1.0.0/24(ro,sync,no_root_squash)
/mnt/repo   10.1.0.0/24(ro,sync,no_root_squash)
```

Apply with:

```bash
exportfs -r
```

Place your kickstart file at `/mnt/ks/ks.cfg` and your ISO or repo under `/mnt/repo`.

### PXE configuration

Update your PXE menu (GRUB/iPXE) entry:

```text
menuentry 'Install Rocky Linux' --class fedora --class gnu-linux --class gnu --class os {
   linuxefi /pxeboot/vmlinuz inst.ks=nfs:10.1.0.15:/mnt/ks/ks.cfg inst.repo=nfs:10.1.0.15:/mnt/repo/ ip=dhcp
   initrdefi /pxeboot/initrd.img
}
```

## Notes

The steps listed in this article apply equally to Rocky Linux 8, 9, and 10.

## References & related articles

[Rocky Linux NFS Server Guide](https://docs.rockylinux.org/guides/file_sharing/nfsserver/)  
[How to Setup a PXE Server on Rocky Linux 9.x](https://kb.ciq.com/article/rocky-linux/rl-pxe-boot-kickstart-file)


PXE Install Fails on Rocky Linux When Kickstart and Repo Share NFS Export


## Introduction

This article addresses an issue surrounding the `python3-devel` package in Rocky Linux 8.10, after upgrading from 8.9. It provides guidance on proper package usage during development and package building processes.

## Problem

When upgrading from Rocky Linux 8.9 to Rocky Linux 8.10, if the `python3-devel` package was previously installed, the user may see the following error after running the `dnf update` command:

```bash
Error:
 Problem: cannot install both platform-python-3.6.8-62.el8_10.rocky.0.x86_64 from baseos and platform-python-3.6.8-56.el8_9.3.rocky.0.x86_64 from @System
  - package python3-devel-3.6.8-56.el8_9.3.rocky.0.x86_64 from @System requires platform-python = 3.6.8-56.el8_9.3.rocky.0, but none of the providers can be installed
  - cannot install the best update candidate for package platform-python-3.6.8-56.el8_9.3.rocky.0.x86_64
  - problem with installed package python3-devel-3.6.8-56.el8_9.3.rocky.0.x86_64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
```

## Resolution

1.- Disable the `devel` repository unless actively building packages. The `devel` repository is intended only for development/build environments. It may contain packages not included in the `base` or `extras` repositories, and is not recommended for general usage.

2.- Use the `platform-python-devel` package instead of `python3-devel`. On upstream Enterprise Linux 8, `python3-devel` is not available as a standard package. Instead, the appropriate development package is `platform-python-devel`, which is included in the `AppStream` repository.

you can use the `--allowerasing` option to move forward with the update or manually remove the `python3-devel` package and install `platform-python-devel` by running:

```bash
dnf install platform-python-devel
```

## Notes

Use recommended macros and alternatives for compatibility. You can check the package info by running:

```bash
rpm -qpi https://download.rockylinux.org/pub/rocky/8/devel/x86_64/os/Packages/p/python3-devel-3.6.8-62.el8_10.rocky.0.x86_64.rpm
```

The `python3-devel` package specifically states:

```bash
It also makes the "python3" and "python3-config" commands available for compatibility with some build systems.
When building packages, prefer requiring platform-python-devel and using
the %{__python3} macro instead, if possible.
```

If the python3 command is needed in this specific case, consider installing the `python36` packages instead.

## References & related articles

[Rocky Linux Documentation, Notes on: Devel](https://wiki.rockylinux.org/rocky/repo/#notes-on-devel)


Python Devel Package Not Available for Rocky Linux 8.10


## Introduction

Rocky Linux ships a default system Python interpreter with each major release. Although the upstream Python project may end support for a given version before the Rocky Linux release reaches end of life, CIQ continues to support the system Python for the full lifecycle of that Rocky Linux release.

This article covers which Python version ships as the default on each major Rocky Linux release, how to find and install alternate Python versions from AppStream, and what the support boundaries are.

## Problem

The default system Python on Rocky Linux may show as end-of-life on the upstream Python release schedule. This can cause confusion about whether the interpreter is still supported and whether security fixes are still being applied. Additionally, some applications or development workflows may require a newer Python version than the system default.

## Resolution

### Default system Python by release

Each major Rocky Linux version ships a specific Python interpreter as the system default:

- Rocky Linux 8: Python 3.6
- Rocky Linux 9: Python 3.9
- Rocky Linux 10: Python 3.12

CIQ supports these versions for the full lifecycle of their respective Rocky Linux release, regardless of the upstream Python project's own end-of-life dates. Security fixes continue to be applied to the system Python as part of the distribution's maintenance cycle.

### Rocky Linux 8: finding and installing alternate versions

Rocky Linux 8 uses AppStream module streams for alternate Python versions. To see what's available:

```bash
dnf module list python*
```

The available module streams are Python 3.6 (default), 3.8, and 3.9. To enable and install an alternate stream:

```bash
dnf module enable python39
dnf install python39
```

The alternate interpreter is available as `python3.9` on the command line. The system `python3` continues to point to the default 3.6 version.

### Rocky Linux 9 and 10: finding and installing alternate versions

Rocky Linux 9 and 10 ship additional Python interpreters as non-modular AppStream packages. These can be installed alongside the system Python without replacing it.

To see which alternate versions are available:

```bash
dnf list available 'python3.*-libs' | grep -oP 'python3\.\d+' | sort -uV
```

As of Rocky Linux 9.8, the available alternate streams are Python 3.11, 3.12, and 3.14. Rocky Linux 10 includes Python 3.14 as an alternate stream.

Install the alternate interpreter using `dnf`. For example, to install Python 3.12 on Rocky Linux 9:

```bash
dnf install python3.12
```

The alternate interpreter is available as `python3.12` on the command line. The system `python3` command continues to point to the default version.

### Using pip with an alternate version

To use pip with any alternate Python interpreter, invoke it through the versioned command:

```bash
python3.12 -m pip install <package>
```

### Support boundaries

CIQ support for Python on Rocky Linux covers the interpreters shipped in BaseOS and AppStream. A few things to keep in mind:

- The system Python is supported for the full Rocky Linux lifecycle
- Alternate AppStream streams have shorter individual lifecycles than the system Python
- Support doesn't extend to third-party packages installed from PyPI
- Python interpreters from EPEL or other third-party repositories aren't covered by CIQ support

## References & related articles

[Rocky Linux release lifecycle](https://wiki.rockylinux.org/rocky/version/)


Python Version Support on Rocky Linux


## Introduction

In high-availability server environments, minimizing downtime is critical. Renaming a network interface often requires a reboot, which can disrupt services. This guide provides methods to rename network interfaces dynamically, ensuring changes take effect without requiring a system restart.

## Problem

Renaming network interfaces without rebooting can be a challenge in Linux systems. Traditional methods typically require a restart to apply changes, which isn't ideal for systems that demand continuous uptime. This limitation can hinder administrators managing critical services or performing live configurations.

## Resolution

### Prerequisites

Access to the `root` user account or a user with escalated privileges.

### Method 1 - `ip link set`

To temporarily change the name of an interface, use the `ip link set` command.

Bring down the network interface that you wish to rename:

```bash
sudo ip link set <INTERFACE_NAME> down
```

Rename the interface:

```bash
sudo ip link set <INTERFACE_NAME> name <NEW_INTERFACE_NAME>
```

Enable the newly renamed interface:

```bash
sudo ip link set <NEW_INTERFACE_NAME> up
```

**Please note that rebooting your server removes the changes.**

### Method 2 - `udev` rules by MAC address

Check under `/etc/udev/rules.d/` that you have a `70-persistent-net.rules` file. If the file isn't there, create it with `touch /etc/udev/rules.d/70-persistent-net.rules`

Confirm the interface you wish to edit by using `ip -brief address show`. You can see a similar output as in the below example:

```bash
$ ip -brief address show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
enp8s0           UP             <PUBLIC_IP>/24
```

Open `/etc/udev/rules.d/70-persistent-net.rules` with your text editor of choice and modify the `NAME` field of the interface you wish to change. A before and after example follows:

Before:

```bash
$ cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="<MAC_ADDRESS_HERE>", NAME="enp8s0"
```

After:

```bash
$ cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="<MAC_ADDRESS_HERE>", NAME="new-name-1"
```

Bring the network interfaces down with `ip link set <INTERFACE_NAME> down`. In this case `enp8s0`:

```bash
ip link set enp8s0 down 
```

Use the `udevadm control` command to reload the `udev` rules and request device events from the kernel with `udevadm trigger`:

```bash
udevadm control --reload-rules; udevadm trigger --subsystem-match=net --action=add
```

Use `ip -brief address show` to confirm that the system has renamed the network interface:

```bash
$ ip -brief address show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
new-name-1       DOWN           <PUBLIC_IP>/24
```

Bring the link back up using `ip link set <NEW_INTERFACE_NAME> up`. Please see the following example:

```bash
ip link set new-name-1 up 
```

Observe that the system renamed the interface and brought it up without requiring a reboot:

```bash
$ ip -brief address show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
new-name-1       UP             <PUBLIC_IP>/24
```

Even if you reboot the machine, the changes remain persistent.

## References & related articles

[`udevadm` man page](https://linux.die.net/man/8/udevadm)  
[`udev` rules introduction](https://opensource.com/article/18/11/udev)


Renaming Network Interfaces Without Rebooting


## Introduction

Losing access to the root account on a Rocky Linux server can be an issue when you need to perform administrative tasks or troubleshooting. You can regain root access by resetting the root password using the GRUB bootloader. This article will guide you through the process, which works for Rocky Linux 8, 9, and 10.

## Resolution

There are two primary methods to reset the root password: using `init=/bin/sh` or `rd.break` at boot time. Both are effective, but `init=/bin/sh` is generally more straightforward and works more reliably across Rocky Linux 8, 9, and 10, including virtual and physical machines. The `rd.break` method is often preferred as well and works in situations where you need to perform more extensive operations within the initial ramdisk environment before the root filesystem is fully mounted, such as dealing with complex LVM configurations or troubleshooting boot time issues.

### Using `init=/bin/sh`

This method is recommended because it is simple, does not require chrooting, and works consistently across most Rocky Linux installations:

- Reboot the server.
- At the GRUB menu, highlight the kernel you wish to boot and press `e` to edit.
- Find the line starting with `linux` or `linux16` or `linuxefi`.
- At the end of this line, add: `init=/bin/sh`
- Press `Ctrl + x` to boot with these parameters.

![GRUB menu with init=/bin/sh highlighted; the target line is the fourth one starting with "linux".](/images/articles/reset-root-password/grub-screen.jpg)

- When the shell prompt appears, remount the root filesystem as read-write:

  ```shell
  mount -o remount,rw /
  ```

- Reset the root password:

  ```shell
  passwd root
  ```

- If SELinux is enabled, ensure proper relabeling on next boot:

  ```shell
  touch /.autorelabel
  ```

- Reboot the system:

  ```shell
  /usr/sbin/reboot -f
  ```

### Using `rd.break`

This method serves as a helpful alternative if you experience difficulties with the primary approach, or if your system uses LVM or a more complex partitioning scheme.

- Reboot the server.
- At the GRUB menu, highlight the kernel and press `e` to edit.
- On the line starting with `linux`, add `rd.break` to the end.
- Press `Ctrl + x` to boot.
- At the `switch_root` prompt, remount the sysroot as read-write:

  ```shell
  mount -o remount,rw /sysroot
  ```

- Change root into the sysroot:

  ```shell
  chroot /sysroot
  ```

- Reset the root password:

  ```shell
  passwd root
  ```

- Relabel for SELinux:

  ```shell
  touch /.autorelabel
  ```

- Exit the chroot and reboot:

  ```shell
  exit
  exit
  ```

## Notes

- Console or vconsole directives in the GRUB boot line (e.g., `console=ttyS0,115200n8`) may need to be removed if you are working in a virtual machine or if you do not see the expected shell prompt.
- If SELinux is enabled and you do not relabel the filesystem, you may encounter login issues after resetting the password.
- The relabel process can take anywhere from a few minutes to over half an hour, depending on your system.
- These procedures require physical or virtual console access to the server.

## References & related articles

[Rocky Linux System Startup Guide](https://docs.rockylinux.org/books/admin_guide/10-boot/)  
[Rocky Linux User Management Guide](https://docs.rockylinux.org/books/admin_guide/06-users/)


How to Reset a Lost Root Password on Rocky Linux


## Introduction

This article addresses the issue where the Sriov configuration script runs manually but fails when executed automatically by systemd. The root cause is identified as a timing or dependency issue.

## Problem

The script `/etc/sriov/setup_dpdk_nic.sh` works when executed manually after a node restart but fails when run automatically by systemd. This indicates a potential timing or dependency issue with the service startup.

## Symptoms

- Script executes successfully when run manually after node restart.
- Script fails when triggered automatically by systemd during the boot process.

## Resolution

Adjust the service unit file to address potential timing and dependency issues. Update the service unit file as follows:

```ini
[Unit]
Description=Sriov Config Service
After=network-online.target systemd-udevd.service
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/etc/sriov/setup_dpdk_nic.sh
RemainAfterExit=yes
TimeoutStopSec=90
#StartLimitInterval=120
#StartLimitBurst=3

[Install]
WantedBy=multi-user.target
```

### Explanation of Changes

1. **[Unit] Section:**
   - **After=network-online.target systemd-udevd.service:** Ensures the service starts only after the network is fully online and systemd-udevd is running. This is crucial for NIC-related configurations.
   - **Wants=network-online.target:** Ensures the network is available when the service runs.

2. **[Service] Section:**
   - **RemainAfterExit=yes:** Keeps the service active after the script completes, aiding in tracking and status reporting.
   - **TimeoutStopSec=90:** Sets a timeout for the stop operation.
   - **StartLimit* Parameters:** Commented out as they control service restart limits, which are less relevant for a oneshot service.

### Benefits

- Ensures the NIC configuration script runs at the correct time during the boot process.
- Improves service status reporting by keeping the service active after script execution.

### Troubleshooting

If the issue persists, provide the output of the following command for further review:

```bash
journalctl -u sriov-configure.service
```

## Root Cause

The problem likely stemmed from the script running before the network was fully online. By adjusting the service dependencies, we ensure the script runs at the appropriate time during the boot sequence.


Resolving Timing and Dependency Issues in Systemd Service Unit for Sriov Configuration


## Introduction

Dual-booting is the practice of installing and using two separate operating systems on one computer. This setup enables you to select which operating system to boot into during startup, offering the flexibility to switch between them as needed.

## Problem

After installing the August 2024 Windows security update, [KB5041585](https://support.microsoft.com/help/5043076) or the August 2024 preview update, you might face issues with booting Linux if you have enabled a dual-boot setup of Windows and Linux on your machine. The result of this issue, is your device might fail to boot Linux and show the following error message:

```shell
Verifying shim SBAT data failed: Security Policy Violation
Something has gone seriously wrong: SBAT self-check failed: Security Policy Violation
```

## Resolution

Please follow the instructions highlighted in [this Microsoft article](https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-23h2#august-2024).

## Root Cause

The August 2024 Windows security and preview updates apply a Secure Boot Advanced Targeting (SBAT) setting to machines that run Windows. This blocks old and vulnerable boot managers. This SBAT update will not be applied to devices where dual-booting is detected. On some machines, the dual-boot detection did not detect some customized methods of dual-booting and applied the SBAT value when it shouldn't have done so.

---
**_IMPORTANT:_** _This known issue only occurs with the installation of the August 2024 security and preview updates. The September 2024 [KB5043076](https://support.microsoft.com/en-us/topic/september-10-2024-kb5043076-os-builds-22621-4169-and-22631-4169-215aad1e-3f3f-44bd-9868-91a2bd450a07) and later security updates do not contain the settings that cause this issue. If you install the September 2024 update, you don’t need to apply the workaround listed above._

---

## Notes

- Disabling Secure Boot is a temporary workaround, but is not recommended for long-term security.
- Ensure your system firmware (BIOS/UEFI) is up to date before applying updates.
- Monitor Windows Update for any future patches addressing this issue.

## References & Related Articles (Optional)

- [Microsoft August 2024 Update Support Article](https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-23h2#august-2024)


Rocky Linux Fails to Boot in a Dual-Boot Setup with Windows


## Introduction

This article provides step-by-step guidance on how to modify screen lock settings in the XFCE desktop environment on Rocky Linux. Whether you need to disable the screen lock for remote sessions or enforce a 10 minute inactivity timeout for local logins, this document offers both GUI and CLI-based solutions. The methods discussed here are aimed at helping you lock or unlock specific settings, ensuring consistency and security across different session types, whether that be local or via a remote session.

## Problem

There are often challenges in managing screen lock preferences, especially when different environments require conflicting behaviors. In some instances, remote sessions should have the lock disabled to avoid disruptions, while local sessions must enforce a 10 minute timeout for security.

## Symptoms

- The screen lock activates automatically at inappropriate times.

- Changes via the GUI or `xfce4-settings-editor` are not immediately apparent, leading to user confusion.

## Resolution

### Prerequisites

- Rocky Linux 8.10 installed on a test machine.

- A fully updated system.

- XFCE desktop installed and properly configured.

- (Optional) NoMachine configured for remote access testing.

### Screenlock configuration methods

#### GUI method

##### Power Manager – System

- Open **Settings → Power Manager → System**.

- Set the **When inactive for** option to **Never** to disable automatic screen locking.

- Ensure that under **Security**, the **Lock screen when system is going to sleep** option is unchecked.

##### Power Manager – Display:

- Open **Settings → Power Manager → Display**.

- Confirm that **Display power management** is set to ON.

- Set **Blank after**, **Put to sleep after**, and **Switch off after** to **Never**.

##### Screensaver Settings:

- Open **Settings → Screensaver**.

- Disable the **Enable Screensaver** option.

- Under **Lock Screen**, uncheck the **Enable Lock Screen** option.

#### CLI method

##### To disable the automatic screen lock for the session:

```bash
xfconf-query -c xfce4-session -p /general/LockCommand -s ""
```

##### To re-enable the lock screen:

```bash
xfconf-query -c xfce4-session -p /general/LockCommand -s "/usr/bin/gnome-screensaver-command --lock"
```

##### Toggle the power management (DPMS-Display Power Management Signaling-controls whether screen blanking runs or not) setting on/off:

```bash
xfconf-query -c xfce4-power-manager -p /xfce4-power-manager/dpms-enabled -T
```

##### Toggle the screen lock on/off:

```bash
xfconf-query -c xfce4-screensaver -p /lock/enabled -T
```

##### Toggle the screensaver on/off:

```bash
xfconf-query -c xfce4-screensaver -p /saver/enabled -T
```

##### To disable the lock screen:

```bash
xfconf-query -c xfce4-screensaver -p /lock/enabled -s false
```

##### To enable the lock screen:

```bash
xfconf-query -c xfce4-screensaver -p /lock/enabled -s true
```

##### Add a 10 minute delay before the screen locks:

```bash
xfconf-query -c xfce4-screensaver -p /lock/saver-activation/delay -s 10
```

##### How to verbosely list settings:

```bash
xfconf-query -c xfce4-power-manager -l -v
xfconf-query -c xfce4-screensaver -l -v
xfconf-query -c xfce4-session -l -v
```

## Root cause

The challenges encountered stem from XFCE’s management of screen lock settings across multiple independent components (Power Manager, Screensaver, and Session settings). These components can lead to conflicting behaviors when trying to restrict or enforce specific lock preferences, especially in environments that require different configurations for remote versus local sessions.

## Notes

Always test changes in a controlled or non-production environment before deployment.

**Tip:** Keep the `xfce4-settings-editor` open during any changes to observe the settings being updated in real-time.

## References & related articles

[XFCE Power Manager Documentation](https://docs.xfce.org/xfce/xfce4-power-manager/start)  
[XFCE Screensaver Configuration Documentation](https://docs.xfce.org/apps/xfce4-screensaver/start)  
[NoMachine for Linux](https://www.nomachine.com)


Managing Screen Lock Preferences in XFCE on Rocky Linux


## Introduction

PyKdump is a powerful extension for the `crash utility`, designed to integrate Python scripting into the crash analysis workflow. It enhances the `crash utility`'s capabilities by providing additional commands and tools that streamline the investigation of `vmcore` dumps.

In particular, PyKdump provides invaluable insight when delving into SCSI device details. Its commands, such as `scsishow --check`, provide additional information on SCSI command statuses, error codes, and device interactions that are not available in the standard `crash utility`.

This article details the installation of the PyKdump extension and recommended SCSI commands to help analyze a `vmcore` dump for a node's filesystem that has reached full capacity.

## Problem

Kernel crashes can occur when a disk is filled to 100%, resulting in a prolonged process halt and eventual panic.

An `sosreport` depending on the time it is taken, sometimes does not reveal filesystem overutilization. CIQ highly recommends to take the `sosreport` as soon as the issue occurs.

If you are unsure as to how to generate an `sosreport`, [please see the full guide available here.](https://kb.ciq.com/article/rocky-linux/rl-generate-an-sosreport)

If the issue cannot be identified in the `sosreport`,  this necessitates further analysis of the `vmcore` dump to pinpoint filesystem-related issues.

## Symptoms

* The system experiences a complete halt of processes for a period of time before the kernel panic ensues.

* Despite the severe impact, the post-crash `sosreport` does not show any evidence of a filesystem filling up, making the root cause from the stance of the `sosreport` difficult to track down.

## Resolution

### Prerequisites
  
* Rocky Linux 8.10

* This article assumes you are familiar with setting up and using the crash utility. For more information, please follow [this guide](https://kb.ciq.com/article/rocky-linux/rl-analysis-of-vmcore-dump-with-crash).

### PyKdump installation

* You will need to install `Python 3.8.20` in order for PyKdump to compile successfully.

* In addition, you also need to build `crash 8.0.5` from source for PyKdump to work.

* Install the `Development Tools` group of packages:

```bash
dnf group install -y "Development Tools"
```

* Please run the below BASH script in order to compile `Python 3.8.20`, `crash` and build `PyKdump`:

```bash
#!/bin/bash

dnf config-manager --set-enabled powertools

dnf install -y readline-devel

dnf install -y texinfo

dnf install -y lzo-devel

git clone git://git.code.sf.net/p/pykdump/code pykdump

wget https://www.python.org/ftp/python/3.8.20/Python-3.8.20.tar.xz

tar -xf ./Python-3.8.20.tar.xz

cd Python-3.8.20/

./configure CFLAGS=-fPIC --disable-shared

cp ../pykdump/Extension/Setup.local-3.8 ./Modules/Setup.local

make

cd ..

wget https://github.com/crash-utility/crash/archive/refs/tags/8.0.5.tar.gz

tar -xf 8.0.5.tar.gz

cd crash-8.0.5/

make lzo

cd ~/pykdump/Extension

./configure -p /root/Python-3.8.20 -c /root/crash-8.0.5

make

cp mpykdump.so ~/mpykdump.so
```

### Pkdump usage

* Move the `crash` binary over to `/usr/bin`:

```bash
sudo cp /path/to/crash-8.0.5 /usr/bin
```

* Start the `crash utility` with a `vmlinux` file and `vmcore` dump selected using this command:

```bash
crash /path/to/vmlinux /path/to/vmcore
```

* Enable the PyKdump extension by running the following in the `crash utility` CLI:

```bash
crash> extend /root/mpykdump.so
Setting scroll off while initializing PyKdump
/root/mpykdump.so: shared object loaded
```

* Here are some recommended commands with which to find more information about `scsi` devices:

### Recommended commands

#### To get a summary of all of your SCSI devices

```bash
scsishow --check

### Summary:

    Task                             Errors/Warnings
    ------------------------------------------------
    SCSI host checks:                0
    SCSI device, command checks:     1
    SCSI target checks:              0

 ** Execution took   0.05s (real)   0.05s (CPU)
```

#### To get the I/O requests pending for a device

```bash
scsishow -r

fa41a307e427e3s0 (10:0:0:1)    timeout: 30000      deadline: 11590928508
Requests found in SCSI layer: 1

 ** Execution took   0.04s (real)   0.04s (CPU)
```

#### To get the last in-flight SCSI commands that were running at the time of the crash

```bash
scsishow -c

scsi_cmnd ff41a507e427e4f8 on scsi_device 0xffa41a307e427e3s0 (10:0:0:1) jiffies_at_alloc: 11590898508
```

#### To check SCSI host adapters for any that are in `busy`, `blocker` or `failed` state

```bash
scsishow -s | grep host

NAME      NAME                   Scsi_Host                shost_data               hostdata                
host0     ahci                   ffa41a307e427e3s0                       0         ff21b221c1429b58
   host_busy           : 0
   host_blocked        : 0
   host_failed         : 0
   host_self_blocked   : 0
   shost_state         : SHOST_RUNNING
```

#### To check for high `IOERR-CNT` I/O error counts

```bash
scsishow -d | awk '{print $1,$NF}' | sort -nrk2

sda 3712821
```

#### To get the `IOREQ-CNT` (I/O Request Count) and the `IODONE-CNT` (I/O Done Count)

```bash
scsishow -d | awk '{print $1,$8,$9}' | sort -nrk2

device / IOREQ-CNT / IODONE-CNT
sdw 10500750 10500750
```

## References & related articles

[PyKdump User Documentation](https://pykdump.readthedocs.io/en/latest/userdoc/index.html)  
[Linux Kernel Crash Book by Igor Ljubuncic](https://www.dedoimedo.com/computers/www.dedoimedo.com-crash-book.pdf)


How to Perform Further SCSI Device Analysis with the PyKdump Crash Extension


## Introduction

Smart cards add an additional layer of authentication to your Linux system. Smart cards have the benefit of being a physical device, that (generally speaking with some exceptions) cannot be reprogrammed once the private keys have been inserted. 

This article will go over how to setup smart card authentication on the MATE desktop environment. The article assumes that the user has a Yubikey / other form of smart card available and has knowledge of setting up a smart card on GNOME. The following method requires the installation of the GNOME desktop environment.

## Problem

You are running Rocky Linux in an enterprise environment, with workstations that have the MATE desktop environment installed on them. You want to add an additional layer of security, by making sure all employees log in using their smart card.

## Resolution

Set up a Rocky Linux 9 System with the MATE desktop. The pre-built [Desktop/Workstation Live Image](https://dl.rockylinux.org/pub/rocky/9/live/x86_64/Rocky-9-MATE-x86_64-latest.iso) is a good option for a smooth install experience.

The GDM lock screen from the GNOME desktop environment is needed for the smart card authentication process. The easiest way to obtain `GDM` (and the GNOME desktop environment as well) is to install the `Workstation` group package with:

```bash
dnf group install "Workstation"
```

* Then enable the GDM service with:

```bash
systemctl enable gdm --force
```

* After that, reboot the machine and the GDM login manager will appear.

* Log into GNOME and configure your smart card.

* Once done, log back out to the GDM login manager.

* Click the little cog icon in the bottom right-hand corner, select the `MATE` option, enter your password, and continue to log in from there.

* To lock your session, run the command below. This will switch the current `MATE` session to the GDM lock screen:

```bash
gdmflexiserver -ls
```

## References & related articles

[Ubuntu MATE Forums - Setup and Locking/Unlocking MATE With a Smart Card](https://ubuntu-mate.community/t/setup-and-locking-unlocking-mate-with-a-smart-card/27461/19)


Enabling Smart Card Support on the MATE Desktop Environment


## Introduction

This article addresses soft lockup issues that can occur during application execution on Rocky Linux.

The recommendations provided here are based on a Slurm environment with high CPU utilization from a Python application, causing CPU lockups.

This guide focuses on Rocky Linux 8.x and 9.x.

## Problem

Soft lockup messages are observed from an application with high CPU utilization.

## Symptoms

System logs from `dmesg` display lockup messages such as `BUG: soft lockup - CPU#18 stuck for 20s! [python3:104938]` during program execution.

`dmesg` also includes `raw_spin_unlock_irqrestore` messages, indicating that a process may be holding a lock for an extended period and causing contention of system resources.

The timing of the soft lockup messages are closely tied to the `watchdog threshold` setting, which can be checked under `/proc/sys/kernel/watchdog_thresh`.

## Resolution

### Improperly configured `/etc/sysconfig/grub` file

- Verify that the `/etc/sysconfig/grub` file is configured correctly with options such as `console=ttyS1` or `console=ttyS1,9600`, and consider increasing the baud rate to `115200` or disabling serial console logging.

### High load average

- High load averages, often exceeding the number of available logical CPU cores, can cause further issues.

- It is worth installing the `sysstat` package with `sudo dnf install -y sysstat` and checking the generated `/var/log/sa` files.

- If the load average is higher than the total number of logical CPU cores available, that can also cause the `soft lockup` messages.

### The watchdog_thresh kernel parameter configuration

- The value set in `watchdog_thresh` determines the threshold interval the kernel uses to decide when a CPU is “stuck.”

- By default, the `watchdog_thresh` value is set to `10` seconds.

- The soft lockup threshold is twice the value of the `watchdog_thresh` parameter.

- The general advice is not to change the default `watchdog_thresh` value, as this can mask hard lockups that occur.

- Verify the current `watchdog_thresh` value with the following command:

```bash
cat /proc/sys/kernel/watchdog_thresh
```

### Application tuning

- Investigate the behavior of the application to identify if it is monopolizing CPU resources, and tweak it to reduce CPU load.

### Tuned profile selection

- Apply the `tuned` profile optimized for high-performance computing workloads by running:

```bash
sudo dnf install -y tuned
sudo systemctl enable --now tuned
sudo tuned-adm profile hpc-compute
sudo tuned-adm active
```

### CPU time limitations

- Limit CPU time for user processes by updating `/etc/security/limits.conf`. An example is below:

```bash
username soft cpu 60  
username hard cpu 120
```

- In the above example, the `soft` limit (this can be moved up or down by the user within the `hard` limit) is `60` seconds and the `hard` limit is enforced by the kernel and is set to `120` seconds. The `hard` limit cannot be increased by the user.

### Vmcore dump analysis

- While configuring `kdump` is outside the scope of this article, once configured, trigger a crash using the following commands:

```bash
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger
```

- By default, the `vmcore` dump will be available under `/var/crash`.

- Going through the `vmcore` dump using the `crash` utility, will also help you understand what is causing the softlockup issues.

## Root cause

Soft lockups occurs when the kernel scheduler is starved and cannot perform its duties within the allotted watchdog threshold.

A high system workload, particularly from a misbehaving application, can saturate CPU resources and prevent task switching from occurring.

System misconfigurations, such as improper serial console settings or not setting the correct `tuned` profile, can also cause lockups to occur.

## References & related articles

[limits.conf man page](https://linux.die.net/man/5/limits.conf)  
[Linux limits.conf man page](https://man7.org/linux/man-pages/man5/limits.conf.5.html)  
[Kernel Crash Dump documentation](https://www.kernel.org/doc/Documentation/kdump/kdump.txt)  
[watchdog_thresh documentation](https://www.kernel.org/doc/html/latest/admin-guide/sysctl/kernel.html#watchdog-thresh)


Diagnosing and Mitigating Soft Lockup Issues on Rocky Linux


## Introduction

When creating ext4 filesystems on loopback devices (image files), you may encounter unexpected behavior where a fully allocated image file becomes sparse after mounting. This article explains why this happens and how to prevent it.

## Problem

When creating an ext4 filesystem on a fully allocated file and then mounting it, some blocks may be unexpectedly optimized or trimmed, potentially resulting in a sparse file. This behavior can be problematic in situations where you need the backing file to remain fully allocated to ensure the mounted filesystem stays robust against ENOSPC (no space left on device) errors on the host filesystem.

## Symptoms

The issue can be observed through the following sequence:

1. Create an allocated file of roughly 5 GB:

   ```bash
   fallocate -l 5368709120 cache.img
   ```

2. Confirm the allocation:

   ```bash
   stat "--printf=%b\n" cache.img
   # Output: 10485784
   ```

3. Create an ext4 filesystem:

   ```bash
   mkfs.ext4 -E nodiscard cache.img
   ```

4. Check allocation again:

   ```bash
   stat "--printf=%b\n" cache.img
   # Output: 10485784
   ```

5. Mount the filesystem:

   ```bash
   mount -o defaults,nodiscard cache.img /mnt/test
   ```

6. After a short period, check allocation:

   ```bash
   stat "--printf=%b\n" cache.img
   # Output: 10321936
   ```

Notice that the number of allocated blocks may decrease after mounting, indicating that the file has become sparse despite using the `nodiscard` option.

## Resolution

To prevent ext4 from sparsifying the loopback file during the first mount, use the following options when creating the filesystem:

```bash
mkfs.ext4 -E nodiscard,lazy_itable_init=0,lazy_journal_init=0 cache.img
```

These options disable the lazy initialization features that defer part of the filesystem creation to the kernel at first mount time, which can cause the sparsification.

## Root Cause

The issue can occur because ext4's default behavior includes lazy initialization of the inode table and journal. When using the default settings:

1. `mkfs.ext4` creates the filesystem but defers some initialization work
2. During the first mount, the kernel completes the initialization
3. As part of this process, certain I/O patterns or optimizations may lead to blocks being treated as sparse, depending on the host filesystem and configuration

The `lazy_itable_init=0` and `lazy_journal_init=0` options force more complete initialization during filesystem creation, helping to prevent deferred work that could result in sparsification.

## Notes

- This behavior is specific to ext4 filesystems. Other filesystems like XFS do not typically exhibit this behavior.
- The `nodiscard` mount option alone may be insufficient to prevent this behavior in all cases.
- While disabling lazy initialization can prevent sparsification, it may increase the time required to create the filesystem.

## References

[Rocky Linux Filesystem Guide](https://docs.rockylinux.org/books/admin_guide/07-file-systems/)  
[ext4 mkfs documentation](https://man7.org/linux/man-pages/man8/mke2fs.8.html)  
[ext4 mount options](https://man7.org/linux/man-pages/man5/ext4.5.html)


Preventing Sparse Allocation in ext4 Loopback Filesystems


## Introduction

When SSH rejects a user's key and falls back to a password prompt, one cause is a group-writable or world-writable home directory. With `StrictModes` enabled (the OpenSSH default), sshd refuses to read `authorized_keys` from a home directory it considers insecure, so key authentication fails even when the key and the `~/.ssh` permissions are correct.

This is one of several possible causes, and a single server-side log line confirms it: `Authentication refused: bad ownership or modes for directory`. If that line is absent, a different problem is at fault and this fix doesn't apply. See the Notes section for the other causes to check.

This applies to Rocky Linux 8, 9, and 10 running OpenSSH with the default `StrictModes yes`.

## Problem

A subset of users can't log in with their SSH keys. The key is present, `~/.ssh` and `authorized_keys` have correct ownership and permissions, and password authentication still works. Copying a working user's `.ssh` permissions doesn't help, because the problem isn't in `.ssh`. It's one level up, on the home directory itself, which is group-writable or world-writable.

This pattern is common on shared systems such as HPC (High Performance Computing) login nodes and Open OnDemand deployments, where home directories are created in bulk and can inherit a group-write bit.

## Symptoms

From the user's point of view, they're prompted for a password when connecting even though their key is installed, while other users on the same system log in with their keys normally.

On the wire, the client offers the key, the server rejects it, and SSH falls back to password authentication. A verbose client trace (`ssh -vvvv localhost`) shows the key being offered and then the method list restarting:

```text
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user01/.ssh/id_rsa RSA SHA256:...
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
...
debug1: Next authentication method: password
user01@localhost's password:
```

The affected user's home directory is group-writable, while a working user's isn't:

```text
# broken user
drwxrwx--- 2 user01 user01 4096 Jun 29 15:10 /home/user01

# working user
drwx------ 2 user02 user02 4096 Jun 29 14:53 /home/user02
```

The server-side sshd log (`journalctl -u sshd` or `/var/log/secure`) confirms the cause:

```text
Authentication refused: bad ownership or modes for directory /home/user01
```

## Resolution

These steps require root or sudo to read the sshd log and adjust another user's home directory.

1. Confirm the cause. The sshd log on the target host shows the rejection for the affected user's home directory. If this line isn't present, key authentication is failing for a different reason and the remaining steps won't help.

   ```text
   Authentication refused: bad ownership or modes for directory /home/user01
   ```

2. Confirm `StrictModes` is enabled (this is the default):

   ```bash
   sshd -T | grep -i strictmodes
   ```

   ```text
   strictmodes yes
   ```

3. Remove group and other write permissions from the affected user's home directory:

   ```bash
   chmod g-w,o-w /home/user01
   ```

4. Confirm the user can now authenticate with their key, then correct the remaining home directories. This targets only the immediate home directories and skips stray files and symlinks. Adjust the base path where home directories actually live, which may not be `/home` on HPC systems:

   ```bash
   find /home -mindepth 1 -maxdepth 1 -type d -exec chmod g-w,o-w {} +
   ```

SSH requires that the home directory, `~/.ssh`, and `authorized_keys` aren't writable by group or other. Target permissions are `0755` or stricter on the home directory (`0700` is common), `0700` on `~/.ssh`, and `0600` on `authorized_keys`.

## Root cause

`StrictModes yes` tells sshd to check the ownership and permissions of the home directory and SSH files before trusting `authorized_keys`. A group-writable or world-writable home directory would let another account in that group add a key and log in as the user, so sshd refuses to use the file. Key authentication then fails and SSH moves to the next allowed method, which presents as a password prompt. Because the loose permission is on the home directory and not inside `.ssh`, matching a working user's `.ssh` permissions doesn't resolve it.

## Notes

**_⚠️ WARNING_** _Disabling `StrictModes` (`StrictModes no` in `/etc/ssh/sshd_config`) also stops the rejection, but it's not recommended. It removes the protection against another user in the same group planting a key in a writable home directory and impersonating the account. Fix the directory permissions instead._

Where home directories are created by automated provisioning or an identity service, check the default umask or the directory-creation logic so new home directories don't come out group-writable.

If the sshd log doesn't show the `bad ownership or modes for directory` message, the cause is elsewhere. Check the permissions on `~/.ssh` (`0700`) and `authorized_keys` (`0600`), whether the correct public key is present in `authorized_keys`, the SELinux context on the home directory (`restorecon -Rv /home/user01`), and whether `PubkeyAuthentication` is disabled in `sshd_config`.

## References & related articles

[OpenSSH sshd_config manual (StrictModes)](https://man.openbsd.org/sshd_config#StrictModes)  
[Unable to SSH to Compute Nodes After Image Update](/article/warewulf/ww-ssh-cluster-key-missing)  
[Enabling Local User Login with Active Directory Authentication Using Kerberos](/article/rocky-linux/rl-local-user-AD-kerberos)  
[How to Enable 2FA-Enabled SSH Log-ins for Rocky Linux](https://ciq.com/blog/how-to-enable-two-factor-authentication-ssh-logins-on-rocky-linux)


SSH Key Auth Fails on Group-Writable Home Directories


## Introduction

On Rocky Linux hosts that get sudo rules from FreeIPA (IdM) through SSSD, a command can show as NOPASSWD in `sudo -l` but still prompt for a password when you run it.

## Problem

`sudo -l` lists the command as NOPASSWD, but running it prompts anyway. On service accounts or hosts with `!visiblepw`, it fails with `sudo: a password is required`.

## Symptoms

```text
sudo -l
    (ALL : ALL) NOPASSWD: /bin/su, tcpdump, top, du, df

sudo tcpdump
sudo: a password is required
```

## Resolution

The rule lists commands by bare name (`tcpdump`) instead of absolute path. Define each command by its full path.

1. Find the absolute path. Run as root, since `/usr/sbin` is often not in a user's `PATH`:

```bash
readlink -f "$(command -v tcpdump)"
# /usr/sbin/tcpdump
```

2. As an IdM admin (`kinit admin`), replace the bare-name entry with the full path:

```bash
ipa sudocmd-add /usr/sbin/tcpdump
ipa sudorule-add-allow-command rule_name --sudocmds=/usr/sbin/tcpdump
ipa sudorule-remove-allow-command rule_name --sudocmds=tcpdump
```

3. Add the no-password option if the rule lacks it (the IdM equivalent of NOPASSWD):

```bash
ipa sudorule-add-option rule_name --sudooption='!authenticate'
```

4. Refresh the client cache and verify:

```bash
sudo sss_cache -R
sudo -l
sudo tcpdump
```

## Root Cause

`sudo -l` prints the rule's stored command string verbatim. At run time, sudo matches the command by absolute path, so a bare name never matches `/usr/sbin/tcpdump`. No NOPASSWD entry applies, so sudo asks for a password.

## If It Still Prompts

- The rule lacks the `!authenticate` option (step 3).
- The rule is assigned to a group, not the user: `ipa sudorule-find --user=<user>` returns nothing. Check each group with `ipa sudorule-find --group=<group>`.
- The SSSD sudo responder is off: confirm `sudo` is in the `services` line of `/etc/sssd/sssd.conf`, then run `authselect enable-feature with-sudo`.
- On sudo 1.8.23 and later, sudo runs PAM even for no-password rules, so a broken PAM stack prompts.

## Notes

After any rule change, run `sudo sss_cache -R` (or restart `sssd`) so the client does not serve stale rules.

List a user's effective rules, including group-inherited ones: `sudo -ll -U user@example.com`.

## References & related articles

[sudoers(5): command matching](https://www.sudo.ws/docs/man/sudoers.man/)

[FreeIPA: Sudo rule management](https://freeipa.readthedocs.io/en/latest/workshop/8-sudorule.html)

[Listing All Sudo Rules for a FreeIPA User](https://kb.ciq.com/article/rocky-linux/rl-list-freeipa-sudo-rules)


Sudo Prompts for a Password Despite a NOPASSWD Rule in FreeIPA


## Introduction

To get the most optimizations and power from your hardware in an Enterprise Linux system, utilization of `TuneD` and `tuned-adm` is paramount.

This guide will explore setting your `tuned` profiles via a configuration file. If you are a System Administrator that deploys pre-configured Rocky Linux images across a fleet of machines, the steps listed here will help you set a pre-configured `tuned` profile.

## Problem

A `tuned profile` can be set via the CLI, for example, with `tuned-adm profile hpc-compute`, however, you're looking to set the profile via a configuration file.

## Resolution

Designate the profile that you want set on the system. In this example, we'll be setting `hpc-compute` as the default profile, as we want to fully utilize the available system resources.

By default, the `tuned` profile that is selected when installing Rocky Linux as a virtual machine is `virtual-guest`. That is the default across Rocky Linux 8, 9, and 10.

For bare metal Rocky Linux installations, the default `tuned` profile across Rocky Linux 8, 9, and 10 is `balanced`.

To set a `tuned` profile in a configuration file, you need to use the `child profile` system. The main profiles are located in `/usr/lib/tuned/`, while the child profiles are located under `/etc/tuned/`. `/etc/tuned/` is also where you can customize profiles.

### Example configuration with hpc-compute

In this example, the bare metal node has `balanced` set as the default profile. We will be changing the default profile to `hpc-compute` and storing the change in a configuration file.

Create the directory:

```bash
mkdir /etc/tuned/balanced/
```

Copy the contents of `/usr/lib/tuned/hpc-compute/tuned.conf` into `/etc/tuned/balanced/tuned.conf`:

```bash
cat /usr/lib/tuned/hpc-compute/tuned.conf > /etc/tuned/balanced/tuned.conf
```

Through the use of `child profiles`, we have now transformed the default `balanced` profile into an `hpc-compute` profile.

Restart the tuned service to apply the new configuration:

```bash
systemctl restart tuned
Observe that the active profile has now been changed to `hpc-compute`:

```bash
cat /etc/tuned/balanced/tuned.conf
#
# tuned configuration
#

[main]
summary=Optimize for HPC compute workloads
description=Configures virtual memory, CPU governors, and network settings for HPC compute workloads.
include=latency-performance

[vm]
# Most HPC applications can take advantage of hugepages. Force them to on.
transparent_hugepages=always

[disk]
# Increase the readahead value to support large, contiguous, files.
readahead=>4096

[sysctl]
# Keep a reasonable amount of memory free to support large mem requests
vm.min_free_kbytes=135168

# Most HPC applications are NUMA aware. Enabling zone reclaim ensures
# memory is reclaimed and reallocated from local pages. Disabling
# automatic NUMA balancing prevents unwanted memory unmapping.
vm.zone_reclaim_mode=1
kernel.numa_balancing=0

# Busy polling helps reduce latency in the network receive path
# by allowing socket layer code to poll the receive queue of a
# network device, and disabling network interrupts.
# busy_read value greater than 0 enables busy polling. Recommended
# net.core.busy_read value is 50.
# busy_poll value greater than 0 enables polling globally.
# Recommended net.core.busy_poll value is 50.
net.core.busy_read=50
net.core.busy_poll=50

# TCP fast open reduces network latency by enabling data exchange
# during the sender's initial TCP SYN. The value 3 enables fast open
# on client and server connections.
net.ipv4.tcp_fastopen=3
```

* In the above example, running `tuned-adm active` will still show the profile as `balanced`, but in fact will be set to the `hpc-compute` configuration instead.

## References & Related Articles

[Rocky Linux tuned man page](https://rockyman.svr.pizza/8.10/tuned/man8/tuned.8.html)
[Rocky Linux tuned-adm man page](https://rockyman.svr.pizza/8.10/tuned/man8/tuned-adm.8.html)


Set the TuneD Profile via a Config File


## Introduction

[`udev`](https://linux.die.net/man/8/udev) provides a dynamic device directory containing only the files for actually present devices. It creates or removes device node files usually located in the `/dev` directory, or it renames network interfaces.

As part of the hotplug subsystem, `udev` is executed if a kernel device is added or removed from the system. On device creation, `udev` reads the `sysfs` directory of the given device to collect device attributes such as label, serial number or bus device number. These attributes may be used as keys to determine a unique name for the device.

On the other hand, RAID devices are virtual devices created from two or more real block devices. This allows multiple devices (typically disk drives or partitions thereof) to be combined into a single device to hold, for example, a single filesystem. Some RAID levels include redundancy and so can survive some degree of device failure. Linux Software RAID devices are implemented through the `md` (Multiple Devices) device driver. More information can be found [at the mdadm man page.](https://linux.die.net/man/8/mdadm)

## Problem

The user may have created `udev` rules to handle the RAID device creation, causing the `md0_sync` process to hang.

## Symptoms

*Please note that the replication of this issue is random, as the race condition does not happen every time. You may have to run the below `mdadm` command a few times, in order to recreate the race condition.*

The following udev rules were used:

```bash
SUBSYSTEM=="block",ACTION=="add|change",KERNEL=="md*",\
ATTR{md/sync_speed_max}="2000000",\
ATTR{md/group_thread_cnt}="64",\
ATTR{md/stripe_cache_size}="8192",\
ATTR{queue/nomerges}="2",\
ATTR{queue/nr_requests}="1023",\
ATTR{queue/rotational}="0",\
ATTR{queue/rq_affinity}="2",\
ATTR{queue/scheduler}="none",\
ATTR{queue/add_random}="0",\
ATTR{queue/max_sectors_kb}="4096"
```

Then to create the RAID device, the following command was run:

```bash
$ mdadm --create --verbose --chunk=128 --level=6 -n 4 /dev/md0 /dev/vda /dev/vdb /dev/vdc /dev/vdd
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: size set to 52395008K
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
```

You can check the status of the RAID creation by checking the `/proc/mdstat` file:

```bash
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 vdd[3] vdc[2] vdb[1] vda[0]
      104790016 blocks super 1.2 level 6, 128k chunk, algorithm 2 [4/4] [UUUU]
      [>....................]  resync =  4.2% (2203904/52395008) finish=3.7min speed=220390K/sec
```

When you check for a second time, you may find the file is empty or the `resync` completion percentage status has not moved.

You could find the following errors in the logs:

```bash
Sep 11 18:39:21 rockylinux.com systemd-udevd[3394]: nvme2n1: /etc/udev/rules.d/udev.rules:53 Failed to write ATTR{/sys/devices/pci0000:60/0000:60:03.3/0000:65:00.0/nvme/nvme2/nvme2n1/queue/max_sectors_kb}, ignoring: Invalid argument
Sep 12 10:22:33 rockylinux.com systemd-udevd[570894]: nvme2n1: /etc/udev/rules.d/udev.rules:53 Failed to write ATTR{/sys/devices/pci0000:60/0000:60:03.3/0000:65:00.0/nvme/nvme2/nvme2n1/queue/max_sectors_kb}, ignoring: Invalid argument
Sep 12 10:43:13 rockylinux.com systemd-udevd[3450]: nvme2n1: /etc/udev/rules.d/udev.rules:53 Failed to write ATTR{/sys/devices/pci0000:60/0000:60:03.3/0000:65:00.0/nvme/nvme2/nvme2n1/queue/max_sectors_kb}, ignoring: Invalid argument
Sep 12 12:21:44 rockylinux.com systemd-udevd[65370]: nvme2n1: /etc/udev/rules.d/udev.rules:53 Failed to write ATTR{/sys/devices/pci0000:60/0000:60:03.3/0000:65:00.0/nvme/nvme2/nvme2n1/queue/max_sectors_kb}, ignoring: Invalid argument
```

The `md0_resync` task will hang with the following logs (*it may cause the kernel to panic and crash, if you have `kernel.hung_task_panic = 1` included in your `/etc/sysctl.conf` file*):

```bash
Sep 10 10:40:38 rockylinux.com kernel: INFO: task md0_resync:2365161 blocked for more than 122 seconds.
Sep 10 10:40:38 rockylinux.com kernel:      Tainted: G S      W6.6.32-120240613.el9.x86_64 #1
Sep 10 10:40:38 rockylinux.com kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 10 10:40:38 rockylinux.com kernel: task:md0_resync      state:D stack:0     pid:2365161 ppid:2      flags:0x00004000
Sep 10 10:40:38 rockylinux.com kernel: Call Trace:
Sep 10 10:40:38 rockylinux.com kernel: <TASK>
Sep 10 10:40:38 rockylinux.com kernel: __schedule+0x222/0x660
Sep 10 10:40:38 rockylinux.com kernel: schedule+0x5e/0xd0
Sep 10 10:40:38 rockylinux.com kernel: md_do_sync+0xcef/0x1030
Sep 10 10:40:38 rockylinux.com kernel: ? sched_clock_cpu+0xf/0x190
Sep 10 10:40:38 rockylinux.com kernel: ? membarrier_register_private_expedited+0xa0/0xa0
Sep 10 10:40:38 rockylinux.com kernel: md_thread+0xb0/0x160
Sep 10 10:40:38 rockylinux.com kernel: ? super_90_load.part.0+0x350/0x350
Sep 10 10:40:38 rockylinux.com kernel: kthread+0xe3/0x110
Sep 10 10:40:38 rockylinux.com kernel: ? kthread_complete_and_exit+0x20/0x20
Sep 10 10:40:38 rockylinux.com kernel: ret_from_fork+0x31/0x50
Sep 10 10:40:38 rockylinux.com kernel: ? kthread_complete_and_exit+0x20/0x20
Sep 10 10:40:38 rockylinux.com kernel: ret_from_fork_asm+0x11/0x20
Sep 10 10:40:38 rockylinux.com kernel: </TASK>
```

## Resolution

To conclude from the above findings, the `udev` rule for the RAID device may be triggering changes too early, leading to a race condition during the `md0_resync` task. Removing this rule fixes the issue.

One potential solution is to use the [`udevadm settle`](https://linux.die.net/man/8/udevadm) approach or introduce a delay to avoid the race condition from occurring.

Alternatively, monitoring the `md0` status before applying device changes via a script could help to ensure the `resync` completes without `udev` interference.

## Root Cause

This issue is related to common `udev` limitations. At the time of a `udev` query, some storage devices might not be accessible. There might be a delay between event generation and processing, especially with numerous devices involved, causing a lag between kernel detection and link availability. External programs invoked by `udev` rules, such as `blkid`, might briefly open the device, making it temporarily inaccessible for other tasks.

## References & related articles

[`udev` man page](https://linux.die.net/man/8/udev)  
[`mdadm` man page](https://linux.die.net/man/8/mdadm)  
[`udevadm` man page](https://linux.die.net/man/8/udevadm)


Udev Rules Causing RAID Device Creation to Hang


## Introduction

[Security Technical Implementation Guides (STIGs)](https://www.titania.com/resources/guides/disa-stig-compliance-explained) produced by the Defense Information Systems Agency (DISA) are heavily implemented in the Enterprise Linux ecosystem. The configurations provided allow for hardening and mitigation against cybersecurity threats.

When DISA STIG is implemented in Rocky Linux, there are occasions where you see messages in `dmesg` about reduced security. This article will explain one such message in `unhashed kernel memory addresses` and if any action is needed to be taken.

## Problem

When looking at `dmesg` you see a warning: *This system shows unhashed kernel memory addresses*.

## Symptoms

The following messages are shown:

```text
[    0.012710] **********************************************************
[    0.012710] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[    0.012711] **                                                      **
[    0.012711] ** This system shows unhashed kernel memory addresses   **
[    0.012711] ** via the console, logs, and other interfaces. This    **
[    0.012712] ** might reduce the security of your system.            **
[    0.012712] **                                                      **
[    0.012712] ** If you see this message and you are not debugging    **
[    0.012712] ** the kernel, report this immediately to your system   **
[    0.012713] ** administrator!                                       **
[    0.012713] **                                                      **
[    0.012713] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[    0.012713] **********************************************************
```

## Root Cause

This is caused by the `slub_debug=P` option being added to the kernel cmdline parameters during boot. This configuration is required by [DISA STIG in Enterprise Linux 8](https://stigviewer.com/stigs/red_hat_enterprise_linux_8/2020-11-25/finding/V-230279)  and [DISA STIG in Enterprise Linux 9](https://stigviewer.com/stigs/red_hat_enterprise_linux_9/2024-12-04/finding/V-257794) to help mitigate [use-after-free based attacks](https://cwe.mitre.org/data/definitions/416.html).

The message does not indicate any issues with the server’s operation and can be safely ignored.

## References & related articles (Optional)

[DISA STIG Explanation](https://www.titania.com/resources/guides/disa-stig-compliance-explained)
[DISA STIG Requirements for Enterprise Linux 8](https://stigviewer.com/stigs/red_hat_enterprise_linux_8/)  
[DISA STIG Requirements for Enterprise Linux 9](https://stigviewer.com/stigs/red_hat_enterprise_linux_9/)  
[DISA STIG SLUB Requirements for Enterprise Linux 8](https://stigviewer.com/stigs/red_hat_enterprise_linux_8/2020-11-25/finding/V-230279)  
[DISA STIG SLUB Requirements for Enterprise Linux 9](https://stigviewer.com/stigs/red_hat_enterprise_linux_9/2024-12-04/finding/V-257794)  
[DISA STIG for Rocky Linux 8.x](https://docs.rockylinux.org/books/disa_stig/disa_stig_part1/)


Unhashed Kernel Memory Error in DMESG


## Introduction

The `sosreport` tool is a key utility used by system administrators and [CIQ Support](https://portal.ciq.com) teams to collect comprehensive diagnostic information from a Rocky Linux system.

When the `sos` command is run, an `sosreport` is generated and gathers configuration details, system logs, and state information in a structured, compressed archive.

This allows a support team to troubleshoot a problem system with detailed logs, without needing the user to manually run multiple commands in order to achieve the same level of log collection.

An `sosreport` can potentially collect a lot of sensitive information about a system and thus knowing what information an `sosreport` gathers, can ease concerns of system administrators when running the `sos` tool.

## Problem

A Rocky Linux node runs into an issue and needs further in-depth analysis by a 3rd party.

## Resolution

To clarify what an `sosreport` collects, below is a detailed layout:

### Hardware information

* DMI tables (`dmidecode`)

* PCI device tree (`lspci`)

* Loaded kernel modules (`lsmod`)

* CPU/Memory statistics (`free`, `numa`, `/proc/cpuinfo`)

* BIOS and firmware configs from `/boot`

### Kernel and system state

* GRUB configuration (`/boot/efi`)

* Uptime, load, and system identity (`uname`, `hostname`, `uptime`)

* Kernel logs (`dmesg`)

* System journal snapshots (`journalctl_--no-pager`)

### Storage and filesystem

* Mounted filesystems (`mount`, `df`)

* LVM information (`vgdisplay`, `/etc/lvm`)

* Disk usage and I/O statistics (`/var/log/sar` if the `sysstat` package is installed and configured)

* `/etc/fstab` and `autofs` configs

### Network configuration

* IP address and routing tables (`ip addr`, `ip route`)

* Network interfaces, drivers, and statistics (`ethtool`)

* `firewalld` and `nftables/iptables` configs

* Snapshots of `/etc/NetworkManager`, `resolv.conf`, `nsswitch.conf`

### Services and daemons

* Enabled systemd units and logs (`systemctl`, journal slices)
  
* Service-specific config directories (`sssd`, `postfix`, `cups`)

* Init scripts and legacy SysV information

### Authentication and security

* PAM configurations (`/etc/pam.d`)

* `sudoers` file, `sshd_config`, SELinux policy state (`sestatus`)

* Audit rules and logs (`/etc/audit`)

### Logging configuration and data

* rsyslog and journald config (`rsyslog.conf`, `journald.conf`)

* Rotated logs config (`/etc/logrotate.d/`)

* Current and previous logs from `/var/log` (depending on if `logrotate` is setup)

### Package management

* Installed RPMs and installation times (`rpm -qa`)

* Repo configs (`/etc/yum.repos.d`, `/etc/dnf`)

### Process and system usage

* Running processes (`ps`, `pstree`)

* Open files (`lsof`)

* Resource limits and environment variables

### Configuration files

* Entire `/etc` tree including symlinked files

* Configuration fragments in `/etc/sysconfig`, `/etc/modprobe.d`, `/etc/security`, etc

### Sosreport plugins and logs

* `sos_commands` contains outputs of all CLI commands executed

* `sos_logs` for plugin-level logging

* `sos.conf`, `groups.d`, `presets.d` reflects plugins used

## Recommendations for environments containing sensitive data

For environments that handle sensitive data, it is recommended to obfuscate the data before the `sosreport` leaves the premises. Please see 
[Generating sosreports on Rocky Linux](https://kb.ciq.com/article/rocky-linux/rl-generate-an-sosreport) under `Obfuscating an sosreport` for more information on how to obfuscate sensitive data in an `sosreport`.

## Root cause

Concerns about the data that `sosreport` collects in environments where sensitive data needs to be handled with care.

## Notes

`sosreport` should be run as root to ensure complete data collection.

Custom plugins or group selections can limit what is captured.

Data collection of an `sosreport` may be expanded or reduced via `/etc/sos/sos.conf` or CLI flags such as `--only-plugins`.

## References & related articles

[sosreport Source Code](https://github.com/sosreport/sos)


What Data is Collected in an sosreport on Rocky Linux


## Introduction

When attempting to install Rocky Linux over a slow or unreliable network connection including through IPMI (Intelligent Platform Management Interface), you may encounter timeout issues that prevent the installer from loading properly.

## Problem

When the installer fails to load within the expected time frame, you may encounter the following error message:

```text
X did not start in the expected time, falling back to text mode.
```

This error occurs because the graphical installer (X server) is unable to start within the default timeout period. This is often caused by slow network speeds, high latency, or interruptions in the connection, which delay the loading of necessary resources. This issue can also occur when a server is configured to use 802.3ad (LACP) for link aggregation but is not configured to support this during the iPXE boot process. As a result, the network interfaces may fail to establish a proper connection, leading to network communication problems. As a result, the installer fails over to text mode, which may not be desired.

## Resolution

If switching to a faster network connection is not an option, there are two primary methods to resolve this issue:

1. **Increase the timeout period** to give the installer more time to load.
2. **Load the installation media into memory**, which bypasses the need to continuously fetch data over the network.

Both methods involve editing the kernel parameters during the installation process.

### Increase timeout

The default timeout for the installer is 60 seconds. You can increase this timeout by modifying the kernel parameters.

1. During the boot process, press `e` to edit the kernel boot options.
2. Locate the line that begins with `linux` or `linuxefi` and contains the kernel parameters.
3. Add the following parameter to the end of the line:

   ```text
   inst.xtimeout=<SECONDS>
   ```

   For example, to set the timeout to 5 minutes (300 seconds), you would add:

   ```text
   inst.xtimeout=300
   ```

4. Press `Ctrl-X` or `F10` to continue booting with the new parameter.

The installer will now wait up to the specified timeout period before falling back to text mode.

### Load into memory

If increasing the timeout does not resolve the issue or if the network connection is highly unstable, you can force the installer to load the entire installation media into memory. This eliminates the need to continuously fetch data over the network during the installation process.

1. As with the previous method, press `e` to edit the kernel boot options.
2. Add the following parameter to the end of the line:

   ```text
   rd.live.ram
   ```

   This option instructs the installer to copy the entire ISO or installation image into memory.

3. Continue with the installation.

Depending on the size of the installation media and the speed of the network, this process may take some time and you will not see any progress or status update.

## Notes

Loading the installation media into memory can require significant RAM. The amount of memory needed depends on the size of the ISO or image and the packages included. For example, a minimal Rocky Linux installation image may require only a few gigabytes of RAM, while a full *DVD* image with additional packages could require significantly more.

If you’re working in a slow or unreliable network environment and need to install or manage multiple servers, it may be worth setting up a [local mirror](https://kb.ciq.com/article/create-local-repo-using-reposync). A local mirror allows you to host the necessary installation files and package updates within your network, significantly reducing dependency on external connections.

## References & related articles

[How to Create a Local Mirror Using `reposync` with CIQ Depot](https://kb.ciq.com/article/create-local-repo-using-reposync)  


X Timeout When Installing Rocky Linux


## Introduction

Multiple UNIX systems utilize the 32-bit `time_t` type, which is part of the Standard C Library (`libc`). Come January 2038, the `time_t` type will run out of bits and therefore be unable to show the current time, affecting countless applications.

This article will dive into what the Y2K38 bug means for Rocky Linux and also quell any fears that the bug will affect the popular operating system.

## Problem

On January 2038, the `time_t` type will not have enough bits available to store the current time, therefore causing multiple 32-bit reliant systems to fail.

An example is an `ext4` filesystem with an inode size of `128`. Once January 2038 hits, this then causes file timestamps to not be correctly recorded.

## Symptoms

32-bit clocks will start to overflow and come back with false values. Example real-world applications that would be impacted are financial forecasts and long-term mortgage interest calculations.

## Resolution

Thankfully, steps have already been taken to mitigate the Y2K38 issue. These include:

* [time_t improvements in the Kernel 5.6 onwards](https://lkml.org/lkml/2020/1/29/355), notably replacing `time_t` with safer alternatives.

**_⚠️ WARNING_** _Caveats do exist however, including ensuring that user space is compiled with the 64-bit version of `time_t` and other considerations._

* [XFS inode timestamps were increased in Kernel 5.10](https://lore.kernel.org/lkml/20201014205059.GD9837@magnolia/) to support dates up till the year 2486.

All current versions of Rocky Linux including 8, 9 and 10 include the above-mentioned patches in their kernels.

* For `ext4` users - if you're running with an inode size of `128` or less, you will need to move the data to a fresh `ext4` partition that has been configured with a higher inode size. It is possible to check the inode size, using the `tune2fs` tool like in the below example:

```bash
tune2fs -l /dev/sdb1 | grep "Inode size"
# In this case, the inode size is at 256, so this ext4 partition is not affected by the Y2K38 bug
Inode size:              256
```

## Summary

The Y2K38 bug does not affect the latest versions of Rocky Linux 8, 9 or 10 at this time.

## References & related articles

[Arnd Bergmann - [GIT PULL] y2038: core, driver and file system changes](https://lkml.org/lkml/2020/1/29/355)  
[Darrick J. Wong - [GIT PULL] xfs: new code for 5.10, part 1](https://lore.kernel.org/lkml/20201014205059.GD9837@magnolia/)  
[François Marier - Upgrading an ext4 filesystem for the year 2038](https://feeding.cloud.geek.nz/posts/upgrading-ext4-filesystem-for-y2k38/)  
[The Register - Linux 5.10 to make Year 2038 problem the Year 2486 problem](https://www.theregister.com/2020/10/19/linux_5_10_y2k38_fixes/)  
[The Register - Need 32-bit Linux to run past 2038?](https://www.theregister.com/2020/01/30/linux_5_6_2038/)


Y2K38 Bug and Rocky Linux


## Introduction

The [Year 2038 Problem (Y2K38)](https://lwn.net/Articles/776435/) is a known issue affecting systems that rely on a 32-bit `time_t` value. On January 19, 2038, this value will overflow, causing time calculations to break on affected systems. While this might seem like a distant issue, some systems already exhibit symptoms, particularly when incorrect timestamps are recorded due to system clock inconsistencies. If files have access or modification timestamps set in the future, beyond a system’s expected date range, licensing mechanisms and other time-sensitive applications may fail. This article provides a resolution for this issue by identifying and correcting future-dated files, ensuring proper time synchronization, and preventing recurrence.

## Problem

System files with modification, access, or creation timestamps set in the future, could cause issues. Such examples include licensing mechanisms to detect system clock discrepancy, leading to errors.

## Symptoms

Running `stat <file>` shows files with the `Access` timestamp set in the future.

```shell
stat /etc/fstab
  File: /etc/fstab
  Size: 2295            Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 67428642    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:etc_t:s0
Access: 2038-01-18 21:14:07.000000000 -0600 # <-- date in 2038
Modify: 2024-08-05 09:53:08.492859120 -0500
Change: 2024-08-05 09:53:08.493859164 -0500
Birth: 2024-08-05 09:53:08.492859120 -0500
```

Applications may fail to start due to a timestamp validation. We have seen problems while hosting a licensed application, observing the error `System clock has been set back`.

## Resolution

Follow these steps to correct the timestamps:

### 1. Identify Files with Future Dates

To locate files with timestamps set in the future, run the following command:

```shell
# Replace YYYY-MM-DD with a realistic cut-off date for your server
find / -newermt "YYYY-MM-DD" -ls
```

Ensure that you review the list before proceeding.

### 2. Update Timestamps on Affected Files

Once you identify the files, you can update their timestamps using the `touch` command to reset the `Access` and `Modify` times to the current dates.

```shell
find / -newermt "YYYY-MM-DD" -exec touch {} \;
```

Alternatively, to set a specific date (e.g., 2023-11-04), use:

```sh
find / -newermt "YYYY-MM-DD" -exec touch -t 202311040000 {} \;
```

#### Handling Symbolic Links

By default, the `touch` command may not update timestamps of symbolic links. To ensure timestamps of symlinks are also modified, use:

```sh
find / -type l -newermt "YYYY-MM-DD" -exec touch -h {} \;
```

According to the [`touch man page`](https://man7.org/linux/man-pages/man1/touch.1.html):

```code
`-h, --no-dereference` affects each symbolic link instead of any referenced file (useful only on systems that can change the timestamps of a symlink).
```

### 3. Verify Timestamps

After applying the changes, verify that timestamps have been corrected:

```shell
find / -ls
```

### 4. Restart the Affected Application

Once timestamps are corrected, restart the application:

```shell
sudo systemctl restart <your_application_service>
```

## Root Cause

This issue is often caused by an incorrect system clock setting or a failing CMOS battery, which can lead to timestamps being improperly set.

To prevent recurrence:

- Check and replace the **CMOS battery** if needed.
- Ensure the **system clock is correctly set**.

## References & Related Articles

- [Year 2038 Problem](https://en.wikipedia.org/wiki/Year_2038_problem)
- [Linux Touch Command Documentation](https://man7.org/linux/man-pages/man1/touch.1.html)


Year 2038 Issue in Rocky Linux Due to Power Outage


## Introduction

CVE-2025-10263 is a race condition in the Linux kernel on Arm processors, in Translation Lookaside Buffer Invalidation (TLBI) handling during memory permission changes. On affected Arm cores, a broadcast TLBI on one processor may complete before the associated memory accesses on another processor are globally observed. A local attacker can use this window to write to memory after the kernel has changed the translation to forbid writes to that location, allowing an unprivileged local user to gain kernel privileges, or a guest VM to write into memory it no longer owns and escalate toward the hypervisor.

This is fundamentally an Arm hardware erratum, addressed in software by a kernel workaround. It affects only arm64 (aarch64) systems. x86_64 systems are not affected.

This article covers Rocky Linux 8, 9, and 10 on arm64, including the CIQ RLC Pro, RLC Pro LTS, and CIQ Linux Kernel (CLK) variants. It explains what is affected, the current patch status, and why there is no software workaround short of the patched kernel.

## Problem

The flaw stems from a documented Arm erratum: completion of a TLBI is not guaranteed to imply completion of affected memory accesses on other processing elements. In the kernel, this means that after a page's translation is changed to remove write permission and the change is broadcast with a TLBI, a store in flight on another core can still land in the now-protected page. An attacker who can arrange the timing can write to memory belonging to a higher privilege level.

The condition requires a multi-core Arm system. The affected Arm cores include the Cortex-X, Cortex-A (recent generations), and Neoverse families, among others. Single-core configurations and non-Arm architectures are not affected.

Treat the following as affected unless they are running a patched kernel (see Patched Kernels below):

- Rocky Linux 8, 9, and 10 on arm64 (aarch64)
- RLC Pro LTS 8.6, 9.2, 9.4, and 9.6 on arm64
- RLC Pro 10 on arm64
- CIQ Linux Kernel (CLK) 6.18 on arm64

Systems that are not affected:

- All x86_64 systems
- Arm systems based on cores not listed in the Arm erratum advisory

## Status

- **Patched kernels are available for RLC Pro LTS 8.6, 9.2, 9.4, 9.6, RLC Pro 10, and CLK 6.18 as of 2026-06-22.** See the Patched Kernels table below for exact versions.
- **There is no software mitigation for this flaw.** It is a hardware erratum that must be worked around in the kernel. There is no module to block and no sysctl or boot parameter that closes the timing window without the kernel fix. The patched kernel is the resolution.
- **x86_64 is not affected.** If you run no arm64 systems, this CVE does not apply to your fleet.
- **Open a support case** if you need help confirming whether a given arm64 host uses an affected Arm core, or to be notified when the patched kernel for your variant is released.

## Patched Kernels

| Variant | Patched Kernel Version | Released |
| --- | --- | --- |
| [RLC Pro LTS 8.6 (arm64)](https://ciq.com/services/long-term-support/) | `kernel-4.18.0-372.32.1+29.1.el8_6_ciq` | 2026-06-22 |
| [RLC Pro LTS 9.2 (arm64)](https://ciq.com/services/long-term-support/) | `kernel-5.14.0-284.30.1+32.1.el9_2_ciq` | 2026-06-22 |
| [RLC Pro LTS 9.4 (arm64)](https://ciq.com/services/long-term-support/) | `kernel-5.14.0-427.42.1+24.1.el9_4_ciq` | 2026-06-22 |
| [RLC Pro LTS 9.6 (arm64)](https://ciq.com/services/long-term-support/) | `kernel-5.14.0-570.60.1+15.1.el9_6_ciq` | 2026-06-22 |
| [RLC Pro 10 (arm64)](https://ciq.com/products/rocky-linux/pro/) | `kernel-6.12.0-211.22.1+2.1.el10_2_ciq` | 2026-06-22 |
| CIQ Linux Kernel 6.18 (arm64) | `kernel-clk6.18-6.18.35-3.1.el9_clk` | 2026-06-22 |

Confirm what is running on a given system with:

```bash
uname -r
uname -m
```

`uname -m` should report `aarch64` for an affected system. On `x86_64` this CVE does not apply.

## Mitigation

There is no software mitigation for CVE-2025-10263. The vulnerability is an Arm hardware erratum, and the only fix is the kernel workaround delivered in a patched kernel. There is no module to unload, no sysctl to set, and no boot parameter that removes the exposure on an affected core.

What you can do in the interim:

- Confirm which of your systems are arm64 and therefore in scope:

  ```bash
  uname -m       # aarch64 = potentially affected
  ```

- On affected arm64 hosts, apply your normal controls for limiting local code execution and untrusted workloads, since the exploit requires a local attacker (or, in virtualized environments, a guest) able to run code on the machine. This reduces opportunity but does not close the underlying flaw.
- Prioritize the affected arm64 hosts for the kernel update as soon as the patched kernel is published.

## Verification

Once the patched kernel is installed and the system rebooted, confirm the running kernel matches (or is newer than) the patched version listed in the Patched Kernels table:

```bash
uname -r
```

On arm64 systems with affected Arm cores, the patched kernel logs the errata mitigation at boot. Check for it with:

```bash
sudo dmesg | grep "Broken broadcast TLBI completion"
```

If the line `CPU features: detected: Broken broadcast TLBI completion` appears, the kernel has recognized the affected core and activated the TLBI workaround. If it does not appear, either the kernel does not contain the fix or the specific Arm core on that host is not in the affected list.

## Resolution

When CIQ publishes the patched kernel for your variant, install it and reboot:

```bash
sudo dnf update kernel*
sudo reboot
```

For RLC Pro LTS variants the patched kernel comes from the long-term support repository with no additional configuration. After reboot, confirm with `uname -r` that the running kernel matches the published patched version.

## Notes

- This is an Arm-only issue. x86_64 systems are not affected by CVE-2025-10263.
- Because it is a hardware erratum worked around in the kernel, vulnerability scanners may flag it on arm64 systems with no available remediation until the patched kernel ships. The accurate status during that window is "affected, fix in progress, no software mitigation," not "ignorable."
- It is unrelated to the Dirty Frag, Fragnesia, and ITScape vulnerabilities. Their mitigations do not apply here.

## Related Articles

- [CVE-2025-10263 (NVD)](https://nvd.nist.gov/vuln/detail/CVE-2025-10263)
- [CVE-2025-10263 (MITRE)](https://www.cve.org/CVERecord?id=CVE-2025-10263)
- [Arm erratum documentation](https://developer.arm.com/documentation/112137)

Mitigating CVE-2025-10263 (Arm TLBI Race) on Rocky Linux 8, 9, and 10 (arm64)


## Introduction

CVE-2026-31431 ("copy.fail") is a local privilege escalation vulnerability in the Linux kernel's AF_ALG AEAD socket interface (`crypto/algif_aead.c`). A reliable public exploit exists with reimplementations in Python, C, Go, and arm64. CIQ has shipped patched kernels for several RLC Pro LTS variants; this article documents both the resolution (install the patched kernel) and a verified boot-parameter mitigation for systems that cannot be updated immediately.

This applies broadly across the Rocky Linux 8, 9, and 10 product family, including the CIQ LTS variants:

- Rocky Linux 8 (all minor versions, e.g. kernel `4.18.0-553.109.1.el8_10` and earlier)
- RLC Pro LTS 8.x (e.g. kernel `4.18.0-553.117.1+2.1.el8_6_ciq` on 8.6 LTS; includes pinned variants such as 8.8)
- Rocky Linux 9 (all minor versions)
- RLC Pro LTS 9.x (pinned variants 9.2, 9.4, and 9.6)
- Rocky Linux 10 (all minor versions)

Confirm by checking the running kernel's config:

```bash
grep CONFIG_CRYPTO_USER_API_AEAD /boot/config-$(uname -r)
```

If the output is `CONFIG_CRYPTO_USER_API_AEAD=y`, the vulnerable algorithm family is built directly into the kernel image, not compiled as a loadable module. This is the default on the Rocky 8, 9, and 10 kernels above.

## Problem

Any local unprivileged user can open an `AF_ALG` socket of type `aead`, bind to an algorithm such as `authencesn(hmac(sha256),cbc(aes))`, and trigger the vulnerability to gain root. Because the AEAD interface is registered automatically at boot via the kernel's initcall mechanism, the attack surface is present on every affected system without any special configuration.

## Status

- **Patched kernels are available** for RLC Pro LTS 9.x variants 9.2, 9.4, and 9.6 as of 2026-05-01. See the Patched Kernels table below for exact versions.
- **Recommended action:** install the patched kernel via `dnf update kernel*` and reboot. See the Installing the Update section for the full procedure, including any required repo or channel setup; after reboot, see the Resolution section for verification steps and how to revert the boot-parameter mitigation if you applied it.
- **If you cannot update yet:** the Mitigation section documents a verified boot-parameter workaround that blocks the public exploit until a patched kernel is in place.
- **Open a support case** if you need help confirming exposure on your fleet, validating the patched kernel against your workload, or tracking patch availability for a Rocky Linux variant not yet listed below.

## Patched Kernels

| Variant | Patched Kernel Version | Released |
|---|---|---|
| [RLC Pro LTS 9.2](https://ciq.com/services/long-term-support/) | `kernel-5.14.0-284.30.1+27.1.el9_2_ciq` | 2026-05-01 |
| [RLC Pro LTS 9.4](https://ciq.com/services/long-term-support/) | `kernel-5.14.0-427.42.1+19.1.el9_4_ciq` | 2026-05-01 |
| [RLC Pro LTS 9.6](https://ciq.com/services/long-term-support/) | `kernel-5.14.0-570.60.1+8.1.el9_6_ciq` | 2026-05-01 |
| [RLC Pro FIPS 9.2](https://ciq.com/products/rocky-linux/pro/) | `kernel-5.14.0-284.30.1+27.1.el9_2_ciq` | 2026-05-01 |
| [RLC Pro FIPS 9.6](https://ciq.com/products/rocky-linux/pro/) | `kernel-5.14.0-570.60.1+8.1.el9_6_ciq` | 2026-05-01 |
| [RLC Pro LTS 8.6](https://ciq.com/services/long-term-support/) | `kernel-4.18.0-372.32.1+24.1.el8_6_ciq` | 2026-05-02 |
| [RLC Pro FIPS 8.10](https://ciq.com/products/rocky-linux/pro/) | `kernel-4.18.0-553.120.1+3.2.el8_10_ciq` | 2026-05-03 |
| [RLC Pro FIPS 8.6](https://ciq.com/products/rocky-linux/pro/) | `kernel-4.18.0-553.117.1+2.1.el8_6_ciq` | 2026-05-04 |
| [RLC Pro Hardened LTS 9.6](https://ciq.com/products/rocky-linux/pro/hardened/) | `kernel-5.14.0-570.60.1+8.1.el9_6_ciq` | 2026-05-04 |
| [RLC Pro LTS 8.10](https://ciq.com/services/long-term-support/) | `kernel-4.18.0-553.120.1+3.2.el8_10_ciq` | 2026-05-05 |
| [RLC Pro 8](https://ciq.com/products/rocky-linux/pro/) | `kernel-4.18.0-553.120.1+3.2.el8_10_ciq` | 2026-05-05 |
| [RLC Pro 9](https://ciq.com/products/rocky-linux/pro/) | `kernel-5.14.0-611.34.1+2.1.el9_7_ciq` | 2026-05-05 |

Confirm what is running on a given system with:

```bash
uname -r
```

If your system reports one of the patched versions above (or newer), the fix is in place and the boot-parameter mitigation can be reverted. See Resolution.

## Installing the Update

### RLC Pro LTS and FIPS Variants

For RLC Pro LTS and FIPS variants other than RLC Pro FIPS 8.10, the patched kernel is available directly from the [long-term support repository](/article/general/ciq-lts-repositories). No additional repository configuration is required:

```bash
sudo dnf update kernel*
sudo reboot
```

### RLC Pro 8, RLC Pro 9, and RLC Pro LTS 8.10

For RLC Pro 8, RLC Pro 9, and the RLC Pro LTS 8.10 variant, the patched kernel is available in the Core Repo. To access it, refresh your DNF metadata and then enable the appropriate depot channel for your variant if you do not have the core repository:

```bash
sudo dnf refresh
```

Then enable the channel that matches your product:

```bash
sudo depot enable lts-8.10   # RLC Pro LTS 8.10
sudo depot enable rlc-pro-8  # RLC Pro 8
sudo depot enable rlc-pro-9  # RLC Pro 9
```

Once the channel is enabled or reenabled, install the update:

```bash
sudo dnf update kernel*
sudo reboot
```

For Rocky Linux community releases, refer to [Rocky Linux Errata](https://errata.build.resf.org/) for patched kernel availability and update instructions.

## Resolution

After the reboot, verify that the running kernel matches the patched version listed in the Patched Kernels table:

```bash
uname -r
```

If you previously applied the boot-parameter mitigation (`initcall_blacklist=algif_aead_init`), remove it now that the fix is in place:

```bash
sudo grubby --update-kernel=ALL --remove-args="initcall_blacklist=algif_aead_init"
sudo reboot
```

After that reboot, confirm the parameter is gone from the active command line:

```bash
grep initcall_blacklist /proc/cmdline
```

The grep should produce no output. The AEAD interface is then restored on the patched kernel, with the upstream fix in place rather than the workaround.

## Mitigation

On all affected EL kernels, `crypto/algif_aead.c` is compiled directly into the kernel image (`CONFIG_CRYPTO_USER_API_AEAD=y`), not as a loadable module (`=m`). That has two practical consequences for anyone trying to mitigate this without a kernel update:

- There is no `algif_aead.ko` for `rmmod` to unload, so the standard "remove the module" approach does not apply.
- An `/etc/modprobe.d/` blacklist has nothing to match against, since the code is part of the kernel image rather than a module the kernel would otherwise load.

The supported way to disable a built-in initcall is the kernel command-line parameter `initcall_blacklist`, which tells the kernel to skip a named initcall function during boot. Adding `initcall_blacklist=algif_aead_init` prevents the AEAD algorithm family from registering, so user-space attempts to bind to AEAD algorithms via AF_ALG fail with `ENOENT`. The same parameter also works on kernels where the code is built as a module, so it is the cleanest single recipe across all the affected variants.

**_⚠️ WARNING_** _Disabling AEAD registration removes a kernel feature that some software relies on. Read the Notes section below and validate against your workload before rolling this out fleet-wide. This is a mitigation, not a fix. Revert it once a patched kernel is installed._

Apply the parameter and reboot:

```bash
sudo grubby --update-kernel=ALL --args="initcall_blacklist=algif_aead_init"
sudo reboot
```

After the reboot, verify the parameter is on the active command line:

```bash
grep initcall_blacklist /proc/cmdline
```

You should see `initcall_blacklist=algif_aead_init` in the output.

Then confirm the kernel actually skipped the AEAD initcall by checking the kernel ring buffer:

```bash
sudo dmesg | grep 'initcall.*algif_aead'
```

You should see a line indicating the initcall was blacklisted at boot. If you do not see it, the kernel command line was not honored. Re-check the cmdline output above and confirm the system was rebooted after `grubby` ran.

Once a patched kernel ships for your variant, follow the Resolution section to install it and revert this mitigation.

## Root Cause

The vulnerability is in `crypto/algif_aead.c` in the AF_ALG AEAD socket interface. The upstream fix is commit `a664bf3d603dc3bdcf9ae47cc21e0daec706d7a5`. The vulnerable code path is reachable on any kernel where `CONFIG_CRYPTO_USER_API_AEAD` is enabled, which is the default on Rocky Linux 8, 9, and 10, and the corresponding LTS kernels.

## Notes

The mitigation has scope beyond just the exploit. Before applying it widely, evaluate the impact on the following:

- **IPsec ESN (Extended Sequence Numbers).** The AEAD subsystem is used by IPsec for ESN support. With AEAD disabled, IPsec connections that negotiated ESN fall back to standard sequence numbers, which means more frequent rekeying. Functionally still works, but rekeying cadence changes.
- **Userspace consumers of AF_ALG AEAD.** `cryptsetup`, `bluez`, `iwd`, and `libkcapi` may open AF_ALG AEAD sockets. Independent testing reported on the oss-security mailing list found inconclusive results with `cryptsetup` (no confirmed breakage, but the researcher noted further testing is warranted). If your workload depends on any of these for AEAD operations, test the mitigation in a non-production environment first.
- **OpenSSL afalg engine.** OpenSSL builds with the afalg engine enabled continue to function for non-AEAD operations. The mitigation only blocks the AEAD algorithm family. The symmetric cipher (`algif_skcipher`) and hash (`algif_hash`) interfaces remain active.
- **Public exploit availability.** The original public exploit is a Python script that requires Python 3.10 or newer. On systems where only an older Python is available the script will not run unmodified, but C, Go, and arm64 reimplementations exist and run with no additional dependencies. The Python version constraint is not a meaningful security boundary. Treat any system with `CONFIG_CRYPTO_USER_API_AEAD=y` as exploitable until patched or mitigated.

## References & related articles

- Public exploit reference: [github.com/theori-io/copy-fail-CVE-2026-31431](https://github.com/theori-io/copy-fail-CVE-2026-31431)
- Vulnerability landing page: [copy.fail](https://copy.fail/)


Mitigating CVE-2026-31431 on Rocky Linux 8, 9, 10, and LTS Variants


## Introduction

CVE-2026-43037 is a flaw in the Linux kernel's `ip6_tunnel` driver, in the ICMP error handler `ip4ip6_err()` for IPv4-in-IPv6 (ip4ip6) tunnels. The handler does not clear the socket buffer control block (`skb2->cb[]`) before reusing the buffer, leaving stale control-block data that the error path then acts on. The upstream fix clears `skb2->cb[]` in `ip4ip6_err()`.

The vulnerable code lives in the `ip6_tunnel` kernel module and is only reachable on systems that have an IPv4-in-IPv6 tunnel configured. This article covers Rocky Linux 8, 9, and 10, CIQ Bridge (CentOS 7.9 extended lifecycle), and the CIQ RLC Pro and RLC Pro LTS variants. It describes what is affected, the current patch status, and how to reduce exposure on systems that use these tunnels until a patched kernel is available.

## Problem

`ip6_tunnel` handles IPv4-in-IPv6 encapsulation (`ip4ip6` / `ip6tnl` tunnel devices). When the kernel receives an ICMP error for traffic carried in such a tunnel, `ip4ip6_err()` processes a cloned skb without first clearing its control block. The stale `cb[]` contents are then interpreted by the error path, which can lead to out-of-bounds handling of the buffer.

The vulnerable path is only present when an IPv4-in-IPv6 tunnel exists on the system, because `ip6_tunnel` is not loaded until such a tunnel device is created. Systems with no ip4ip6/ip6tnl tunnel do not load the module and do not expose this code.

Treat the following as affected when they have an IPv4-in-IPv6 tunnel configured, unless they are running a patched kernel (none is available yet, see Status):

- Rocky Linux 8, 9, and 10 community releases
- RLC Pro LTS 8.6, 9.2, 9.4, and 9.6
- RLC Pro 8 (el8_10), RLC Pro 9 (el9_8), and RLC Pro 10 (el10_2)
- RLC Pro FIPS variants on 8.6
- CIQ Bridge (CentOS 7.9)

This is a networking flaw, not architecture-specific. It affects x86_64 and arm64 alike.

## Status

- **No patched kernel is available yet, as of 2026-06-11.** CIQ is tracking the fix across the affected Rocky Linux 8, 9, and 10, LTS, FIPS, and CIQ Bridge kernels. This article will be updated with exact patched kernel versions and a Patched Kernels table once builds are released.
- **Recommended interim action:** if you do not use IPv4-in-IPv6 tunnels, confirm `ip6_tunnel` is not loaded and, optionally, block it from loading (see Mitigation). If you do use these tunnels, plan to install the patched kernel as soon as it is published, since the module-block mitigation will break your tunnels.
- **Architecture:** both x86_64 and arm64 are affected.
- **Open a support case** if you need help confirming exposure on your fleet, assessing tunnel impact, or tracking patched kernel availability for a specific CIQ variant.

## Patched Kernels

No patched kernels are available yet. CIQ will populate the table below once builds are released.

| Variant | Patched Kernel Version | Released |
| --- | --- | --- |
| RLC Pro LTS 8.6 | _pending_ | _pending_ |
| RLC Pro LTS 9.2 | _pending_ | _pending_ |
| RLC Pro LTS 9.4 | _pending_ | _pending_ |
| RLC Pro LTS 9.6 | _pending_ | _pending_ |
| RLC Pro 8 (el8_10) | _pending_ | _pending_ |
| RLC Pro 9 (el9_8) | _pending_ | _pending_ |
| RLC Pro 10 (el10_2) | _pending_ | _pending_ |
| RLC Pro FIPS 8.6 | _pending_ | _pending_ |
| CIQ Bridge (CentOS 7.9) | _pending_ | _pending_ |

Confirm what is running on a given system with:

```bash
uname -r
```

## Mitigation

The exposure is conditional: the vulnerable code is only reachable when an IPv4-in-IPv6 tunnel is configured. The right mitigation depends on whether you use these tunnels.

### First, determine whether the module is in use

Check whether `ip6_tunnel` is loaded and whether any ip4ip6/ip6tnl tunnel devices exist:

```bash
lsmod | grep -E '^ip6_tunnel'
ip -d link show type ip6tnl
```

If `lsmod` shows nothing and `ip -d link show type ip6tnl` lists no devices, the system is not currently exposed through this path. If a tunnel device is listed, the module is in use and a module block will break it.

### If you do not use IPv4-in-IPv6 tunnels: block the module

On systems that do not use ip4ip6/ip6tnl tunnels, prevent the module from loading so the vulnerable handler cannot be reached.

***⚠️ WARNING*** *This blocks all IPv4-in-IPv6 (`ip4ip6` / `ip6tnl`) tunneling. Do not apply it on any system that uses these tunnels. It does not affect ordinary IPv6 traffic, GRE, WireGuard, or IPsec; it only affects the `ip6_tunnel` encapsulation driver. Validate against your network configuration before applying fleet-wide.*

The vulnerable handler, `ip4ip6_err()`, lives in `ip6_tunnel.ko`. With the module blocked from loading, the function cannot be reached.

Install the modprobe override:

```bash
echo "install ip6_tunnel /bin/false" | sudo tee /etc/modprobe.d/cve-2026-43037.conf
```

If the module is already loaded but no tunnel is active, unload it so the override takes effect immediately:

```bash
sudo rmmod ip6_tunnel
```

If `rmmod` reports the module is in use, a tunnel device still exists. Do not force the unload on a production system; identify and remove the tunnel during a maintenance window, or wait for the patched kernel instead.

### If you do use IPv4-in-IPv6 tunnels

There is no software mitigation that preserves the tunnels. Plan to install the patched kernel as soon as it is published. In the interim, limit who can send traffic to the tunnel endpoints to your trusted networks, since the vulnerable handler is driven by ICMP errors associated with tunneled traffic.

## Verification

If you applied the module block, confirm the module is not loaded:

```bash
lsmod | grep -E '^ip6_tunnel'
```

No output means the module is not currently loaded. Confirm the override is in place and honored:

```bash
cat /etc/modprobe.d/cve-2026-43037.conf
sudo modprobe -n -v ip6_tunnel
```

`modprobe -n -v` should print `install /bin/false`. If it resolves to a `.ko` path, re-check that the file is present and readable.

## Resolution

When CIQ publishes the patched kernel for your variant, install it and reboot.

On Rocky Linux 8, 9, and 10 and the RLC Pro LTS and FIPS variants, the patched kernel comes from the configured CIQ repositories with no additional setup:

```bash
sudo dnf update kernel*
sudo reboot
```

On CIQ Bridge (CentOS 7.9), use `yum` instead, with the patched kernel from the CIQ Bridge repository:

```bash
sudo yum update kernel*
sudo reboot
```

After reboot, confirm with `uname -r` that the running kernel matches the published patched version.

If you applied the module block and you actually need IPv4-in-IPv6 tunnels, remove the override after updating:

```bash
sudo rm /etc/modprobe.d/cve-2026-43037.conf
sudo modprobe ip6_tunnel
```

## Notes

- This is a conditional-exposure vulnerability. A system with no IPv4-in-IPv6 tunnel does not load `ip6_tunnel` and does not expose the vulnerable handler. This is the key fact for triaging scanner hits against this CVE.
- It is unrelated to the Dirty Frag and Fragnesia page-cache corruption vulnerabilities. The ESP module-block mitigations for those CVEs do not address this one, and blocking `ip6_tunnel` does not address those.
- CIQ Bridge (CentOS 7.9) is affected and is being tracked alongside the Rocky Linux variants.

## Related Articles

- [CVE-2026-43037 (NVD)](https://nvd.nist.gov/vuln/detail/CVE-2026-43037)
- [CVE-2026-43037 (MITRE)](https://www.cve.org/CVERecord?id=CVE-2026-43037)
- [Upstream kernel CVE announcement](https://lore.kernel.org/linux-cve-announce/2026050102-CVE-2026-43037-0346@gregkh/T)

Mitigating CVE-2026-43037 (ip6_tunnel) on Rocky Linux 8, 9, 10, CIQ Bridge, and LTS Variants


## Introduction

CVE-2026-46215 is a local privilege escalation vulnerability in the Linux kernel's DRM (Direct Rendering Manager) subsystem. The `DRM_IOCTL_GEM_CHANGE_HANDLE` ioctl moves a GEM (Graphics Execution Manager) buffer object to a caller-chosen handle number, and its handler does not reconcile the handle-to-object mapping atomically. The handler allocates the new handle in the IDR (`idr_alloc`) and removes the old one (`idr_remove`) in separate steps. In the window between them the object holds two live IDR entries, the old handle and the new one, while its `handle_count` still reads 1. A `DRM_IOCTL_GEM_CLOSE` racing on the old handle drops what looks like the last handle reference, decrements `handle_count` to 0, and frees the object while the surviving new handle still points at it. Any subsequent use of that handle thus is a use-after-free exploit.

A public proof-of-concept and technical writeup are available. The exploit turns the use-after-free into a passwordless root account with a data-only technique that needs no memory-corruption gadgets. It reclaims the freed object into a `kmalloc` slab sprayed with pipe buffers, leaks a kernel pointer to defeat KASLR, bypasses the Dirty Pipe fix, and overwrites the page cache for `/etc/passwd` to add a passwordless UID 0 entry.

The scoping detail that matters for CIQ customers is where the vulnerable `change_handle` ioctl came from. It was introduced upstream in kernel 6.18, and it was also backported into the RHEL 9 and RHEL 10 kernel trees. That produces two exposure routes. The el9_8 and el10_2 backport exposes both the RLC Pro (9 and 10) and the Rocky Linux (9.8 and 10.2) kernels, and CIQ Linux Kernel 6.18 carries the feature natively until it is updated to a build that disables it. Most other CIQ kernels never carried the feature and are not affected. On every affected CIQ kernel the fix disables the ioctl outright (see Status).

This article describes what is affected, the current patch status, how to confirm exposure on a given host, how to install the patched kernel or apply an interim mitigation, and how to verify the result.

## Problem

The flaw is reachable through the DRM render node. Both `DRM_IOCTL_GEM_CHANGE_HANDLE` and `DRM_IOCTL_GEM_CLOSE` are flagged `DRM_RENDER_ALLOW`, and `drm_ioctl_permit()` gates them on nothing else: no DRM master, no `DRM_AUTH`, no capability. Any process that can open `/dev/dri/renderD*` can call them. Two consequences are worth calling out during triage.

No graphical session is required. Render-node access is usually granted by `systemd-logind` on desktops, which is the source of most confusion around this CVE, but the bug is triggered by a plain `ioctl()` on a render node with no X11, Wayland, compositor, or logged-in graphical session running. The public proof-of-concept runs headless in a QEMU guest with `virtio-gpu`.

The render node is world-accessible by default. On stock Rocky Linux and RHEL 9/10, `50-udev-default.rules` ships `/dev/dri/renderD*` as `GROUP=render MODE=0666`, so any local UID can open it. The `card*` nodes are seat- and uaccess-gated and do need a graphical login, which is why the render node, not `card0`, is the headless attack path.

The exploitable drivers are those that register GEM plus a render node (`DRIVER_GEM | DRIVER_RENDER`): `virtio_gpu` (common in cloud and virtualized guests), and bare-metal GPU drivers such as `amdgpu`, `i915`, `nouveau`, `radeon`, `vmwgfx`, and `xe`. NVIDIA's proprietary stack is a partial carve-out, since the GEM-create path at default `modeset=0` is unproven and default CUDA containers expose only `/dev/nvidia*`.

Because the exposure depends on both an affected kernel and an openable render node, the highest real-world risk is on multi-tenant or shared hosts where untrusted local users can run code: virtualized guests using `virtio-gpu`, and GPU or AI compute servers where users legitimately hold render-node access.

Treat the following as affected unless running a patched kernel (see Status):

- RLC Pro 9 and Rocky Linux 9.8, via the RHEL 9 backport
- RLC Pro 10 and Rocky Linux 10.2, via the RHEL 10 backport
- CIQ Linux Kernel 6.18, which carries the feature natively (builds older than `6.18.36-1` are affected)

The following CIQ kernels are **not affected**, because the `change_handle` feature was never present in them:

- RLC Pro 8 and RLC Pro LTS 8.6
- RLC Pro LTS 9.2 and RLC Pro LTS 9.6 (5.14-based, but before the RHEL backport)
- CIQ Bridge 7.9
- CIQ Linux Kernel 6.12 (the feature was introduced upstream in 6.18)

### Confirming exposure on a specific host

First confirm the running kernel is on an affected line and older than the patched build:

```bash
uname -r
```

An `el9_8` build older than `5.14.0-687.29.1`, or an `el10_2` build older than `6.12.0-211.37.1`, is affected, on both RLC Pro (which adds a `+2.1...el9_8_ciq` or `+2.1...el10_2_ciq` suffix) and Rocky Linux. A CIQ Linux Kernel 6.18 build older than `6.18.36-1` is also affected. Any other CIQ kernel (LTS 8.6/9.2/9.6, RLC Pro 8, Bridge 7.9, CIQ Linux Kernel 6.12) is not affected by this CVE.

Then check whether a render node exists and which driver backs it, since exposure requires an openable render node bound to an affected driver:

```bash
ls -l /dev/dri/
lsmod | grep -E 'virtio_gpu|amdgpu|i915|nouveau|radeon|vmwgfx|^xe'
```

If there is no `renderD*` node under `/dev/dri/` (common on stock cloud VMs with no GPU, for example default AWS Nitro or Azure `hyperv_drm`, and on GCP instances with no `/dev/dri`), the attack path is not present regardless of kernel version. If a `renderD*` node exists, check its permissions:

```bash
stat -c '%n %a %G' /dev/dri/renderD* 2>/dev/null
```

On a default install this reports mode `666` and group `render`, meaning any local user can open it. That is the exposed configuration.

## Status

- **Patched kernels are available** for every affected line: RLC Pro 9 and 10, Rocky Linux 9.8 and 10.2, and CIQ Linux Kernel 6.18. See the Patched Kernels table below for exact versions. On all of them the fix disables the `change_handle` ioctl by pointing it at `drm_invalid_op`, so the ioctl now returns `-EINVAL` instead of running the vulnerable handle swap. Every affected CIQ kernel ships the full five-commit `change_handle` fix chain that ends in that disable, and there is no variant that keeps the ioctl live behind a narrower race fix.
- **One update clears the whole chain.** The `change_handle` fix chain's RPM changelog names four CVEs: `CVE-2026-46215` (the use-after-free), `CVE-2026-52966`, `CVE-2026-53145`, and `CVE-2026-23149` (a warning-hardening change). The same patched kernel resolves all four, so a scanner citing any of them on an affected line is cleared by the same update.
- **All other CIQ kernels are not affected** (the feature is absent), and no action is required on them for this CVE.
- **Recommended action:** on any affected host, install the patched kernel and reboot (see Installing the Update). RLC Pro 9/10 and Rocky Linux 9.8/10.2 update via `dnf update kernel*`, and CIQ Linux Kernel 6.18 hosts update to `6.18.36-1` or newer. If you cannot update immediately, see Mitigation for interim steps.
- **Open a support case** if you need help confirming exposure on a specific host or validating a patched kernel against your workload.

## Patched Kernels

| Variant | Patched Kernel Version | Released |
| --- | --- | --- |
| [RLC Pro 9](https://ciq.com/products/rocky-linux/pro/) | `kernel-5.14.0-687.29.1+2.1.el9_8_ciq` | 2026-07-24 |
| [RLC Pro 10](https://ciq.com/products/rocky-linux/pro/) | `kernel-6.12.0-211.37.1+2.1.el10_2_ciq` | 2026-07-25 |
| CIQ Linux Kernel 6.18 | `6.18.39-1` | 2026-07-20 |

For Rocky Linux 9.8 and 10.2, the fixed kernels ship through the Rocky Linux errata. Refer to [Rocky Linux Errata](https://errata.build.resf.org/) for the exact published build.

Confirm what is running on a given system with:

```bash
uname -r
```

If your system reports one of the patched versions above (or newer), the fix is in place and any interim mitigation can be reverted. See Resolution.

## Installing the Update

### RLC Pro 9 and 10

The patched kernel comes from the CIQ repositories your variant already uses, with no additional setup:

```bash
sudo dnf update kernel*
sudo reboot
```

If your RLC Pro variant draws the kernel from the Core Repo through depot channels, refresh metadata and enable the channel that matches your product before updating:

```bash
sudo dnf clean all && sudo dnf makecache
sudo depot enable rlc-pro-9   # use rlc-pro-10 on RLC Pro 10
sudo dnf update kernel*
sudo reboot
```

### CIQ Linux Kernel 6.18

Update to `6.18.36-1` or newer through your configured CLK repository, then reboot. Any 6.18 build from `6.18.36-1` onward carries the fix:

```bash
sudo dnf update kernel*
sudo reboot
```

For Rocky Linux community releases, refer to [Rocky Linux Errata](https://errata.build.resf.org/) for patched kernel availability and update instructions.

## Mitigation

> **Warning:** The steps below are interim measures for hosts that cannot be patched immediately. Each one trades off GPU or display functionality. The fix is the patched kernel in Installing the Update. Test in a non-production environment before applying fleet-wide, and revert once the patched kernel is installed.

The exposure requires an unprivileged user to open a render node bound to an affected driver. The two options below each remove one side of that condition. Pick based on whether the host actually needs GPU or display functionality.

### Option 1: Restrict access to the render node

By default `/dev/dri/renderD*` is world read/write (`0666`). Restricting it to the `render` group removes access from arbitrary local UIDs while leaving GPU access intact for accounts you place in that group. This does not fix the kernel bug; it narrows who can reach it. Use this on hosts that use the GPU only through specific accounts.

**Step 1: install the `udev` rule.**

```bash
cat <<'EOF' | sudo tee /etc/udev/rules.d/99-cve-2026-46215.rules
# Interim CVE-2026-46215 mitigation: remove world access to DRM render nodes
SUBSYSTEM=="drm", KERNEL=="renderD*", GROUP="render", MODE="0660"
EOF
sudo udevadm control --reload-rules
sudo udevadm trigger --subsystem-match=drm
```

**Step 2: confirm the new mode and grant access to the accounts that need it.**

```bash
stat -c '%n %a %G' /dev/dri/renderD*
# e.g. usermod -aG render <account>   # for a service account that runs GPU/compute work
```

This leaves the flaw reachable by members of the `render` group, so it only helps where those members are trusted. On a host where every local user must have render access, this option provides no protection; use Option 2 or patch instead.

### Option 2: Block the affected driver module

On a headless VM or server that does not need a display or GPU compute, preventing the affected driver from loading removes the render node entirely. The most common case is `virtio_gpu` on a virtualized guest with no graphical workload.

> **Warning:** Blocking the driver removes the GPU/DRM device. On a guest that uses `virtio-gpu` for its console this can affect the graphical console. Only apply where the host is genuinely headless and does not need the device. Do not use this option on GPU or AI compute nodes, since it removes the functionality they exist to provide.

**Step 1: install the modprobe override.**

```bash
# Replace virtio_gpu with the driver reported by the lsmod check above if different
echo "install virtio_gpu /bin/false" | sudo tee /etc/modprobe.d/cve-2026-46215.conf
```

**Step 2: clear any resident copy of the module.**

If the module is already loaded and not in use, unload it so the override takes effect immediately:

```bash
sudo rmmod virtio_gpu
```

If `rmmod` reports the module is in use, something is actively using the GPU or console framebuffer. Do not force the unload on a production host; schedule the change for a maintenance window, or install the patched kernel instead.

## Verification

Use these checks to confirm an interim mitigation is active. For confirming the patched kernel is running, see Resolution.

### After applying Option 1 (render node permissions)

```bash
stat -c '%n %a %G' /dev/dri/renderD*
```

The mode should read `660`, not `666`. Any local user outside the `render` group can no longer open the node.

### After applying Option 2 (driver block)

Confirm the module is not loaded:

```bash
lsmod | grep virtio_gpu
```

No output means the module is not currently loaded. Confirm the modprobe override is in place:

```bash
sudo modprobe -n -v virtio_gpu
```

This should print `install /bin/false`. If it resolves to a `.ko` path, re-check `/etc/modprobe.d/cve-2026-46215.conf` and confirm it is present and readable. If the module still appears in `lsmod` after a reboot, the override is not catching the early boot load path; confirm the conf file is present and readable, rebuild the initramfs to include the modprobe config, and reboot again.

## Resolution

After installing the patched kernel and rebooting, confirm the running kernel matches a patched version from the Patched Kernels table:

```bash
uname -r
```

If your system reports one of the patched versions (or newer), the fix is in place.

The one behavior change the fix introduces is that `DRM_IOCTL_GEM_CHANGE_HANDLE` now returns `-EINVAL` instead of running. The ioctl was added for a niche checkpoint/restore path and has no real userspace consumers, so the impact is negligible. Normal GPU, compute, and display workloads are unaffected.

If you applied either mitigation, revert it once the patched kernel is running:

```bash
# Revert Option 1 (render node permissions)
sudo rm /etc/udev/rules.d/99-cve-2026-46215.rules
sudo udevadm control --reload-rules
sudo udevadm trigger --subsystem-match=drm

# Revert Option 2 (driver block)
sudo rm /etc/modprobe.d/cve-2026-46215.conf
sudo modprobe virtio_gpu   # or the driver you blocked
```

## Notes

- **Scanner hits.** A vulnerability scanner flagging the kernel on an affected line (RLC Pro 9/10, Rocky Linux 9.8/10.2, or CIQ Linux Kernel 6.18 older than `6.18.36-1`) is accurate. A scanner may cite any of the four CVEs the fix chain carries, `CVE-2026-46215`, `CVE-2026-52966`, `CVE-2026-53145`, or `CVE-2026-23149`, and the same patched kernel resolves all four. Real-world risk is highest where untrusted local users can run code and a render node is present and openable: `virtio-gpu` guests, shared GPU/AI compute nodes, CI runners. A host with no `/dev/dri/renderD*` node has no attack path here even on an affected kernel.
- **Backport versus native.** The affected 5.14 (el9_8) and 6.12 (el10_2) kernels are vulnerable because the `change_handle` feature was backported into them, not because of their base version. That is why RLC Pro LTS 9.2 and 9.6 (also 5.14-based) and CIQ Linux Kernel 6.12 are not affected; they never received the backport. CIQ Linux Kernel 6.18 is the exception, since it carries the feature natively because `change_handle` was introduced upstream in 6.18.
- **Not a display-only issue.** The exploit does not need a desktop, compositor, or graphical login. Assume any affected host with an openable render node and local users is reachable.
- The mitigations are interim. Install the patched kernel and revert the workaround as soon as the build is applied.

## References & related articles

- [CVE-2026-46215 (NVD)](https://nvd.nist.gov/vuln/detail/CVE-2026-46215)
- [CVE-2026-46215 (MITRE)](https://www.cve.org/CVERecord?id=CVE-2026-46215)
- Related CVEs in the same `change_handle` fix chain, resolved by the same patched kernel: [CVE-2026-52966](https://nvd.nist.gov/vuln/detail/CVE-2026-52966), [CVE-2026-53145](https://nvd.nist.gov/vuln/detail/CVE-2026-53145), [CVE-2026-23149](https://nvd.nist.gov/vuln/detail/CVE-2026-23149)
- Public technical writeup: [cyberstan.co.uk/drm-lpe-linux](https://cyberstan.co.uk/drm-lpe-linux/)
- Upstream fix commit (definitive, disables the ioctl): [1a4f03d22fb6](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1a4f03d22fb655e5f192244fb2c87d8066fcfca2)
- Upstream race fix commit: [5e28b7b94408](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5e28b7b94408897e41c63477aabc9e1db439bc8c)


CVE-2026-46215 (DRM GEM change_handle Use-After-Free) on Rocky Linux and RLC Pro Kernels


## Introduction

CVE-2026-46331 ("Pedit COW") is a local privilege escalation vulnerability in the Linux kernel's traffic control (`tc`) packet-editing action, `act_pedit` (`net/sched/act_pedit.c`). The packet-edit action function, `tcf_pedit_act()`, computes the copy-on-write (COW) range for `skb_ensure_writable()` once, before the per-key loop, using the precomputed `tcfp_off_max_hint`. That hint does not account for the runtime header offset added by typed keys, so part of the target write region is edited without a proper copy-on-write. The result is an out-of-bounds write that can corrupt page-cache memory and be leveraged for root.

Red Hat rates this as **Important** (CVSS 3.1 base score **7.8**, vector `CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H`, CWE-787). A public proof-of-concept was released within roughly a day of the CVE being assigned, so treat exposed systems as exploitable until patched or mitigated.

This article covers Rocky Linux 8, 9, and 10 and the CIQ RLC Pro and RLC Pro LTS variants. It describes what is affected, current patch status, how to confirm exposure, and how to reduce risk until a patched kernel is installed.

## Problem

`act_pedit` is the `tc` action that rewrites bytes in packet headers/payloads. To exploit the flaw, an actor with `CAP_NET_ADMIN` configures a qdisc/filter with a `pedit` action whose typed keys push the effective write offset past the region that `tcf_pedit_act()` made writable. Because `skb_ensure_writable()` was only asked to COW up to `tcfp_off_max_hint` — which omits the per-key runtime header offset — the subsequent write lands outside the copied region, an out-of-bounds write into adjacent (potentially page-cache-backed) memory. This is the classic CWE-787 primitive that exploit chains turn into local root.

Two facts drive triage:

- **Capability requirement.** Creating the qdisc/filter requires `CAP_NET_ADMIN`. On a default system that means root. However, an **unprivileged** user can obtain `CAP_NET_ADMIN` inside a user+network namespace (`unshare -Urn`) when unprivileged user namespaces are enabled — which is the default on Rocky Linux 8, 9, and 10. That is the path the public PoC uses, and it is what makes this a meaningful unprivileged-to-root escalation rather than a root-only issue.
- **Module reachability.** The vulnerable code lives in `act_pedit.ko`. The module is auto-loaded the first time a `pedit` action is created, including from inside a namespace, so "we don't use `tc pedit`" is not on its own a guarantee that the module cannot be loaded.

This is a networking/kernel flaw, not architecture-specific. It affects x86_64 and arm64 alike. RHEL 7 and earlier (and CIQ Bridge / CentOS 7.9) are not affected by this code path.

Treat the following as affected unless they are running a patched kernel (see Status):

- Rocky Linux 8, 9, and 10 community releases
- RLC Pro LTS 8.6, 9.2, and 9.6 (kernel branches CIQLTS-8.6, CIQLTS-9.2, CIQLTS-9.6)
- RLC Pro 8 (el8_10), RLC Pro 9 (el9_8), and RLC Pro 10 (el10_2) (kernel branches RLC-8, RLC-9, RLC-10)
- RLC Pro FIPS and Hardened variants built on the affected kernels

## Status

- **Fix status:** The upstream fix was posted to the netdev list on 2026-05-16 and has landed in mainline and the affected stable branches; Red Hat has shipped errata (RHSA-2026:27704, bulletin RHSB-2026-008). For reference, the Red Hat fixed kernels are `kernel-4.18.0-553.136.1.el8_10` (RHEL 8), `kernel-5.14.0-687.17.1.el9_8` (RHEL 9), and `kernel-6.12.0-211.26.1.el10_2` (RHEL 10).
- **CIQ kernels:** Patched kernels are available for all supported CIQ variants. CIQ LTS 8.6, LTS 9.2, and LTS 9.6 shipped 2026-06-30. RLC Pro FIPS 8.6, RLC Pro 8, RLC Pro 9, and RLC Pro 10 shipped 2026-07-01. See the Patched Kernels table below for exact NVRs.
- **Recommended action:** install the patched kernel for your variant via `dnf update kernel*` and reboot (see Resolution). If you cannot update immediately, see Mitigation for two verified interim workarounds.
- **Open a support case** if you need help confirming exposure on your fleet or validating a patched kernel against your workload.

## Patched Kernels

| Variant | Patched Kernel Version | Released |
| --- | --- | --- |
| [RLC Pro LTS 8.6](https://ciq.com/services/long-term-support/) (CIQLTS-8.6) | `kernel-4.18.0-372.32.1+30.1.el8_6_ciq` | 2026-06-30 |
| [RLC Pro LTS 9.2](https://ciq.com/services/long-term-support/) (CIQLTS-9.2) | `kernel-5.14.0-284.30.1+33.1.el9_2_ciq` | 2026-06-30 |
| [RLC Pro LTS 9.6](https://ciq.com/services/long-term-support/) (CIQLTS-9.6) | `kernel-5.14.0-570.60.1+16.1.el9_6_ciq` | 2026-06-30 |
| [RLC Pro FIPS 8.6](https://ciq.com/products/rocky-linux/pro/) (FIPS-8.6) | `kernel-4.18.0-553.137.1+2.1.el8_6.ciqfips` | 2026-07-01 |
| [RLC Pro 8](https://ciq.com/products/rocky-linux/pro/) (RLC-8, el8_10) | `kernel-4.18.0-553.137.1+2.1.el8_10_ciq` | 2026-07-01 |
| [RLC Pro 9](https://ciq.com/products/rocky-linux/pro/) (RLC-9, el9_8) | `kernel-5.14.0-687.17.1+2.1.el9_8_ciq` | 2026-07-01 |
| [RLC Pro 10](https://ciq.com/products/rocky-linux/pro/) (RLC-10, el10_2) | `kernel-6.12.0-211.26.1+2.1.el10_2_ciq` | 2026-07-01 |

Confirm what is running on a given system with:

```bash
uname -r
```

If your system reports one of the patched versions above (or newer), the fix is in place and any interim mitigation can be reverted. See Resolution.

## Confirming Exposure

Check whether the module is loaded and whether any `pedit` actions are configured:

```bash
lsmod | grep -E '^act_pedit'
tc actions ls action pedit
```

If `lsmod` shows nothing and `tc actions ls action pedit` lists no actions, the vulnerable code is not currently active — but note that the module can still be auto-loaded later (including from inside an unprivileged namespace), so this is a point-in-time check, not a guarantee of safety.

Check whether unprivileged user namespaces are enabled, since that is the path that turns this into an unprivileged escalation:

```bash
# Rocky/RHEL 8 (Debian-style toggle is absent; use the max count)
sysctl user.max_user_namespaces
# Quick functional test: succeeds (returns to a shell) if unprivileged userns is allowed
unshare -Urn true && echo "unprivileged user namespaces ENABLED"
```

## Mitigation

The right mitigation depends on whether you use `tc pedit` and whether you rely on unprivileged user namespaces (e.g. rootless Podman). The two options below can be used independently or together.

### Option 1 — Block the act_pedit module (for systems that do not use tc pedit)

***⚠️ WARNING*** *This disables the `tc pedit` packet-editing action entirely. Do not apply it on any system that uses `pedit` for traffic shaping or header rewriting. It does not affect ordinary networking, other `tc` actions, GRE, WireGuard, or IPsec. Validate against your network configuration before applying fleet-wide. This is a mitigation, not a fix — revert it once a patched kernel is installed.*

Prevent the module from loading so the vulnerable function cannot be reached:

```bash
echo "install act_pedit /bin/false" | sudo tee /etc/modprobe.d/cve-2026-46331.conf
```

If the module is already loaded but no `pedit` action is active, unload it so the override takes effect immediately:

```bash
sudo rmmod act_pedit
```

If `rmmod` reports the module is in use, a `pedit` action still exists. Do not force the unload on a production system; remove the offending `tc` action during a maintenance window, or install the patched kernel instead.

### Option 2 — Restrict unprivileged user namespaces (defense-in-depth)

The unprivileged-to-root path depends on an unprivileged user obtaining `CAP_NET_ADMIN` inside a user namespace. Disabling unprivileged user namespaces closes that path while leaving legitimate root use of `tc pedit` intact.

***⚠️ WARNING*** *This breaks rootless containers (Podman/Buildah run as a non-root user) and any other workload that relies on unprivileged user namespaces. Only apply where those workloads are absent. Revert once a patched kernel is installed.*

```bash
echo "user.max_user_namespaces = 0" | sudo tee /etc/sysctl.d/cve-2026-46331.conf
sudo sysctl --system
```

After applying, the functional test from Confirming Exposure should fail:

```bash
unshare -Urn true || echo "unprivileged user namespaces DISABLED"
```

## Verification

If you applied the module block (Option 1), confirm the module is not loaded and the override is honored:

```bash
lsmod | grep -E '^act_pedit'
cat /etc/modprobe.d/cve-2026-46331.conf
sudo modprobe -n -v act_pedit
```

No `lsmod` output means the module is not currently loaded. `modprobe -n -v act_pedit` should print `install /bin/false`. If it instead resolves to a `.ko` path, re-check that the override file is present and readable.

## Resolution

When the patched kernel for your variant is available, install it and reboot.

On Rocky Linux 8, 9, and 10 and the RLC Pro LTS, FIPS, and Hardened variants, the patched kernel comes from the configured CIQ or BaseOS repositories with no additional setup:

```bash
sudo dnf update kernel*
sudo reboot
```

If your RLC Pro variant draws the kernel from the Core Repo via depot channels, refresh metadata and enable the channel that matches your product before updating:

```bash
sudo dnf clean all && sudo dnf makecache
sudo depot enable rlc-pro-8   # RLC Pro 8   (use rlc-pro-9 for RLC Pro 9)
sudo dnf update kernel*
sudo reboot
```

After the reboot, confirm the running kernel matches the published patched version:

```bash
uname -r
```

If you applied either mitigation and you actually need the affected functionality, revert it once the patched kernel is running:

```bash
# Revert Option 1 (module block)
sudo rm /etc/modprobe.d/cve-2026-46331.conf
sudo modprobe act_pedit

# Revert Option 2 (user namespaces)
sudo rm /etc/sysctl.d/cve-2026-46331.conf
sudo sysctl --system
```

## Root Cause

The vulnerability is in `tcf_pedit_act()` in `net/sched/act_pedit.c`. The function calls `skb_ensure_writable()` once before the per-key loop, sizing the writable/COW region from `tcfp_off_max_hint`. That hint is computed without the runtime header offset that typed keys add at execution time, so for a crafted key the actual write offset exceeds the region that was made writable — an out-of-bounds write past the copy-on-write boundary (CWE-787). The upstream fix moves `skb_ensure_writable()` inside the per-key loop, where the real write offset for each key is known, and adds overflow checking on the offset arithmetic so an oversized offset is rejected rather than written.

## Notes

- **Triage tip for scanner hits.** Exposure requires the ability to create a `pedit` action (`CAP_NET_ADMIN`). The practical unprivileged path is unprivileged user namespaces, which are enabled by default on Rocky 8/9/10. A scanner flagging the kernel version is accurate; the real-world risk is highest where untrusted local users can run code and user namespaces are unrestricted (multi-tenant nodes, shared dev hosts, CI runners).
- **OpenShift.** OpenShift documentation rates this as Low severity because the `act_pedit` module is not loaded by default in that environment.
- **Not the same as the page-cache fragmentation CVEs.** Although this is also a page-cache corruption primitive, it is unrelated to the Dirty Frag / Fragnesia ESP issues. The `act_pedit` mitigations here do not address those, and their mitigations do not address this one.
- Mitigations are interim. Install the patched kernel and revert the workaround as soon as a build is available for your variant.

## References & related articles

- [CVE-2026-46331 (NVD)](https://nvd.nist.gov/vuln/detail/CVE-2026-46331)
- [CVE-2026-46331 (MITRE)](https://www.cve.org/CVERecord?id=CVE-2026-46331)
- [Upstream patch on netdev (LKML)](https://lore.kernel.org/netdev/20260516162825.1480113-1-rollkingzzc@gmail.com/)


Mitigating CVE-2026-46331 (act_pedit / "Pedit COW") on Rocky Linux 8, 9, 10, and LTS Variants


## Introduction

CVE-2026-53264 is a use-after-free in the Linux kernel traffic control action layer (`net/sched/act_api.c`). When a traffic control action is looked up by index, `tcf_idr_check_alloc()` finds it in the action `idr` and raises its reference count while holding only the RCU read lock. A concurrent path that drops the last reference removes the action from the `idr` and frees it immediately, with no RCU grace period. If the two overlap, the lookup raises a reference count on memory that has already been returned to the allocator.

A public writeup describing the flaw, along with a proof of concept, has been published. This article does not assess whether any particular Rocky Linux or RLC build can be exploited, and no proof of concept was run against a CIQ kernel to produce it.

This article covers Rocky Linux 9 and 10 and the CIQ RLC Pro, RLC Pro LTS, RLC Plus, RLC Pro AI, and RLC Pro Hardened variants. It describes what is affected, current patch status, how to confirm exposure, and how to reduce risk until a patched kernel is available.

## Problem

Two ingredients are needed for the race. One thread must be able to reach the action lookup, and another must be able to free an action at the same moment. The kernel normally prevents this by holding the RTNL lock across the whole operation, which serialises every path that touches an action.

There is one exception. Filter operations (`RTM_NEWTFILTER`, `RTM_DELTFILTER`, `RTM_GETTFILTER`) are registered so they can run without RTNL, but they only actually do so when both the queueing discipline and the classifier are flagged as safe to run unlocked. In every affected branch, `cls_flower` is the only classifier carrying that flag, and `sch_ingress` (which provides the `ingress` and `clsact` queueing disciplines) is the only provider of an unlocked queueing discipline.

That is what makes the mitigation below work. Remove `cls_flower` and no classifier qualifies for the unlocked path, so the filter handlers fall back to taking RTNL, and the racing pair can no longer form.

Two facts drive triage:

- **Capability requirement.** Configuring a queueing discipline and filter requires `CAP_NET_ADMIN`. On a default system that means root. An unprivileged user can obtain `CAP_NET_ADMIN` inside a user and network namespace (`unshare -Urn`) when unprivileged user namespaces are enabled, which is the default on Rocky Linux 9 and 10 and on every RLC variant checked for this article.
- **Module auto-loading.** Both `cls_flower` and `act_gact` are built as modules and are auto-loaded the first time a matching filter or action is created, including from inside a namespace. "We do not use `tc`" is not by itself a guarantee that the modules cannot be loaded.

This is a networking and kernel flaw and is not architecture-specific.

Treat the following as affected unless they are running a patched kernel:

- Rocky Linux 9.8 and 10.2 community releases
- RLC Pro LTS 8.6, 9.2, and 9.6
- RLC Pro 9 and RLC Pro 10
- RLC Plus 10.2, RLC Pro AI 9.8, and RLC Pro Hardened 9.6

The following are **not** affected. Their version of `tcf_idr_check_alloc()` holds the `idrinfo` lock across both the lookup and the reference count increment, so the racing window does not exist:

- RLC Pro LTS 8.10 and 9.4
- RLC Pro 8

## Status

- **Fix status:** The upstream fix is commit `5057e1aca011`, `net/sched: act_api: use RCU with deferred freeing for action lifecycle`. It defers the free to an RCU callback so the lookup can no longer observe freed memory. It does not change the lookup itself.
- **CIQ kernels:** Patched kernels are available for RLC Pro LTS 9.6 and RLC Pro Hardened 9.6. The two CIQ Linux Kernel channels also carry the fix. Patched kernels for the remaining affected variants are in progress (see the table below).
- **Recommended action:** install the patched kernel for your variant if one is available. For variants where the fix is still pending, apply the module mitigation below if the affected hosts do not rely on `flower` filters.
- **Open a support case** if you need help confirming exposure on your fleet or deciding whether the mitigation is safe for your workload.

## Patched Kernels

| Variant | Patched Kernel Version | Status |
| --- | --- | --- |
| RLC Pro LTS 8.6 | Not yet published | Pending |
| RLC Pro LTS 9.2 | Not yet published | Pending |
| RLC Pro LTS 9.6 | `kernel-5.14.0-570.60.1+20.1.el9_6_ciq` | Available |
| RLC Pro Hardened 9.6 | `kernel-5.14.0-570.60.1+20.1.el9_6_ciq` | Available |
| RLC Pro 9 (el9_8) | Not yet published | Pending |
| RLC Pro 10 (el10_2) | Not yet published | Pending |
| CIQ Linux Kernel 6.12 | `kernel-clk6.12-6.12.98-1.1.el9_clk` | Available |
| CIQ Linux Kernel 6.18 | `kernel-clk6.18-6.18.40-1.1.el9_clk` | Available |

For Rocky Linux community releases, check [Rocky Linux Errata](https://errata.rockylinux.org/) for patched kernel availability.

## Confirming Exposure

Check whether your kernel builds the relevant pieces as modules. On every affected variant checked, both are modules, which is what makes the mitigation applicable:

```bash
grep -E '^CONFIG_(NET_CLS_FLOWER|NET_ACT_GACT|NET_CLS_ACT)=' /boot/config-$(uname -r)
```

Expected output on an affected host:

```text
CONFIG_NET_ACT_GACT=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_CLS_FLOWER=m
```

`CONFIG_NET_CLS_ACT=y` means the action framework itself is built into the kernel and cannot be blocked by any module rule. Only the classifier and the individual actions are modules.

Check whether unprivileged user namespaces are available, since that is how an unprivileged user reaches `CAP_NET_ADMIN`:

```bash
sysctl user.max_user_namespaces
unshare -Urn true && echo "unprivileged userns available"
```

Check whether the modules are currently resident. This matters for the mitigation, as explained in the next section:

```bash
lsmod | grep -E 'cls_flower|act_gact'
```

## Mitigation

Two options. The first is targeted at this flaw. The second is broader defence in depth. Neither replaces the patched kernel.

### Option 1: Block the `cls_flower` and `act_gact` modules

Suitable for systems that do not use `flower` filters. See the warning below before applying this to a host running Open vSwitch, anything layered on it, or `tc`-based traffic shaping.

Create `/etc/modprobe.d/blacklist-cve-2026-53264.conf`:

```text
blacklist act_gact
blacklist cls_flower
install act_gact /bin/false
install cls_flower /bin/false
```

The two `install` lines are the load-bearing part. The kernel requests these modules by name rather than through an alias, and a `blacklist` line on its own does not stop a by-name request.

**This file does not take effect until the modules are unloaded or the host is rebooted.** A `modprobe.d` rule has no effect on a module that is already resident. If `cls_flower` or `act_gact` were loaded when the file was written, the host remains exposed until it reboots. This was reproduced on every variant tested. Reboot the host, or if a reboot is not possible and no traffic control configuration is in use, remove the filters and unload the modules:

```bash
sudo tc qdisc del dev <interface> clsact
sudo rmmod cls_flower
sudo rmmod act_gact
```

⚠️ **Removing `cls_flower` is not free.** Open vSwitch uses `flower` filters for its [hardware offload path](https://docs.openvswitch.org/en/latest/howto/tc-offload/), anything layered on Open vSwitch inherits that, and `tc`-based filtering and shaping can use `flower` directly. On a host that uses any of them, this drop-in will break that functionality. Confirm what the host actually does before rolling this out, and test on a non-production host first.

### Option 2: Restrict unprivileged user namespaces (defence in depth)

The published attack path relies on obtaining `CAP_NET_ADMIN` inside an unprivileged user namespace. Disabling them closes that path for unprivileged local users. It does not address the flaw for a process that already holds `CAP_NET_ADMIN`.

```bash
echo 'user.max_user_namespaces = 0' | sudo tee /etc/sysctl.d/99-disable-userns.conf
sudo sysctl --system
```

⚠️ This breaks rootless containers (Podman and Apptainer) and anything else that relies on user namespaces. Only apply it where nothing on the host uses them.

## Verification

After a reboot, confirm the override is being honoured:

```bash
modprobe -n -v cls_flower
modprobe -n -v act_gact
```

Both should print:

```text
install /bin/false
```

Confirm neither module can be loaded explicitly, and that neither is resident:

```bash
sudo modprobe cls_flower ; echo "rc=$?"
lsmod | grep -E 'cls_flower|act_gact' ; echo "resident=$?"
```

The `modprobe` should return non-zero and `lsmod` should match nothing.

Confirm the auto-load path is closed. On an unmitigated host the following succeeds and loads both modules. On a mitigated host it fails:

```bash
sudo tc qdisc add dev lo clsact
sudo tc filter add dev lo ingress protocol ip flower action gact drop
```

Expected output on a mitigated host:

```text
Error: TC classifier not found.
We have an error talking to the kernel
```

Clean up the test queueing discipline afterwards:

```bash
sudo tc qdisc del dev lo clsact
```

Note that the action framework remains reachable through other classifiers and actions. With the drop-in in place, a filter using the `u32` classifier with a `mirred` action still installs. This is expected. Those classifiers all run under RTNL, so they cannot form the racing pair, but it does mean the mitigation contains the flaw rather than removing the code.

## Resolution

Install the patched kernel for your variant when it is published, then reboot:

```bash
sudo dnf update kernel*
sudo reboot
```

Confirm the running kernel afterwards:

```bash
uname -r
```

Once a patched kernel is running, the module drop-in can be removed:

```bash
sudo rm /etc/modprobe.d/blacklist-cve-2026-53264.conf
sudo reboot
```

For hosts that can move to a CIQ Linux Kernel channel, the 6.12 and 6.18 kernels listed above already carry the fix.

## Root Cause

`tcf_idr_check_alloc()` performs its lookup under `rcu_read_lock()` and then calls `refcount_inc_not_zero()` on the action it found. The teardown path, `__tcf_action_put()`, drops the last reference, removes the action from the `idr` under the `idrinfo` lock, and then frees it synchronously through `tcf_action_cleanup()` and `free_tcf()`, which ends in a bare `kfree()`. Because the free is not deferred to an RCU grace period, a reader that has already obtained the pointer can raise a reference count on freed memory.

The upstream fix defers that free so it cannot land while a reader still holds the RCU read lock. In branches that are not affected, the reader holds the `idrinfo` lock across both the lookup and the reference count increment, which excludes the teardown path outright.

## Notes

- The `act_gact` half of the mitigation blocks only the one action the published proof of concept uses. Every other action module reaches the same framework code. The `cls_flower` half is what closes the race, because it removes the only classifier that can run a filter operation without RTNL.
- `CONFIG_NET_CLS_ACT=y` on all affected variants, so the action framework cannot be blocked with a module rule.
- Blocking `sch_ingress` instead would also close the unlocked path, since it is the only provider of an unlocked `qdisc`, but it is more disruptive because it removes the `ingress` and `clsact` queueing disciplines entirely.

## References & related articles

- [Mitigating CVE-2026-46331 on Rocky Linux 8, 9, 10, and LTS Variants](/article/security-advisories/cve-2026-46331-mitigation)
- [CVE-2026-53264 (NVD)](https://nvd.nist.gov/vuln/detail/CVE-2026-53264)


Mitigating CVE-2026-53264 (Kernel Traffic Control Use-After-Free) on Rocky Linux and RLC Variants

## Introduction

CVE-2026-53361 and CVE-2026-72137 are two Linux kernel flaws in the networking stack. Both carry an Important severity rating, both have a local attack vector, and both have public exploit code available. Among CIQ maintained kernels, only RLC Pro 10 is affected, and a single kernel update addresses both.

CVE-2026-53361, publicly referred to as BadGarbage, is a race condition in the `af_unix` garbage collector. `unix_schedule_gc()` set the `gc_in_progress` flag rather than `unix_gc()` itself, so two concurrent callers could leave the flag false while collection was still running. `unix_peek_fpl()` relies on that flag to avoid confusing the collector during `MSG_PEEK`. Public exploit code demonstrates a use-after-free leading to local privilege escalation and container escape.

CVE-2026-72137 is a double free in the `xfrm` NAT keepalive path. `nat_keepalive_send()` freed the keepalive `skb` on any error from the IPv4 or IPv6 send helper, but once `ip_build_and_send_pkt()` or `ip6_xmit()` has taken ownership the networking stack may already have consumed the buffer. Freeing it a second time corrupts kernel memory. Public exploit code demonstrates local privilege escalation.

## Problem

CVE-2026-53361 was introduced by upstream commit `8b90a9f819dc` ("af_unix: Run GC on only one CPU.") and is fixed by commit `d82ba05263c`. CVE-2026-72137 was introduced by upstream commit `f531d13bdfe3` ("xfrm: support sending NAT keepalives in ESP in UDP states") and is fixed by commit `226f4a490d1`. Both vulnerable code paths are recent additions to the upstream kernel, which is why older product lines are out of scope.

The following CIQ product variant is affected. Treat it as exposed until it is running a patched kernel listed in the Patched Kernels section:

- RLC Pro 10

The following are not affected, because the vulnerable code is not present in their kernels:

- CIQ Bridge 7.9
- RLC Pro 8 and RLC Pro 9
- RLC Pro LTS 8.6, LTS 9.2, and LTS 9.6

The following already carry both upstream fixes:

- CIQ Linux Kernel LT 6.12
- CIQ Linux Kernel 6.18

CLK builds predating those fixes are vulnerable. If you are running a CLK kernel and are unsure whether it includes the patches, open a support case to confirm the specific build.


For community Rocky Linux releases, refer to [errata.rockylinux.org](https://errata.rockylinux.org/) for patch availability.


## Status

- **RLC Pro 10 is the only CIQ variant currently known to be affected.**
- **A patched RLC Pro 10 kernel is available.** See the Patched Kernels table below and update at the earliest opportunity.
- **Public exploit code exists for both CVEs.** Prioritize affected systems accordingly.
- Both flaws require local access. Neither is remotely exploitable on its own.
- **Open a support case** if you need help assessing exposure or tracking patched kernel availability.

## Patched Kernels

| Variant | Patched Kernel Version | Released |
| --- | --- | --- |
| [RLC Pro 10](https://ciq.com/products/rocky-linux/pro/) | `kernel-6.12.0-211.49.1+2.1.el10_2_ciq` | 2026-09-04 |

Confirm what is running on a given system with:

```bash
uname -r
```

## Confirming Exposure

Exposure is determined by the installed kernel package, not by workload configuration.

An RLC Pro 10 system is affected until it runs a patched kernel. Identify the running kernel with:

```bash
uname -r
```

Compare that against the Patched Kernels table above. Any RLC Pro 10 system running a kernel older than the version listed there is affected.

Two cautions on scoping this by configuration:

CVE-2026-53361 is in the core `af_unix` garbage collector, which is exercised by any workload that passes file descriptors over Unix domain sockets. There is no configuration that takes a system out of scope.

CVE-2026-72137 sends an ESP-in-UDP NAT keepalive, so it may appear that only hosts running IPsec are exposed. That is not a safe assumption. A local attacker can create the required `xfrm` state inside their own user and network namespace, so the absence of host IPsec configuration does not rule out exploitation. Do not use existing `xfrm` configuration to classify an affected system as unexposed.

## Mitigation

No configuration change mitigates either flaw. The patched kernel is the resolution.

Because both flaws require local code execution, the practical interim control is limiting who and what can run code on affected hosts. Systems with interactive users, CI/CD runners, shared application accounts, multi-tenant container hosts, or any workload capable of executing untrusted code should be patched first.

## Verification

After updating and rebooting, verify the running kernel:

```bash
uname -r
```

Compare the output against the Patched Kernels table above. If the running kernel matches or is newer than the patched version listed for your variant, the fixes are in place.

## Resolution

1. Update the kernel with `sudo dnf update kernel*`
2. Reboot into the new kernel
3. Verify the running kernel with `uname -r`

If the update does not appear, run `sudo dnf clean all && sudo dnf makecache` to refresh repository metadata.

## Notes

- CVE-2026-53361 scores CVSS 7.8 (`AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H`) and CVE-2026-72137 scores CVSS 7.0 (`AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H`). The difference is attack complexity.

## Related Articles

[CVE-2026-53361 on NVD](https://nvd.nist.gov/vuln/detail/CVE-2026-53361)  
[CVE-2026-72137 on NVD](https://nvd.nist.gov/vuln/detail/CVE-2026-72137)  
[Upstream fix for CVE-2026-53361](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d82ba05263c69fa2437fe93e4e561cc40f4c03af)  
[Upstream fix for CVE-2026-72137](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=226f4a490d1a938fc838d8f8c46a4eca864c0d78)

Mitigating CVE-2026-53361 and CVE-2026-72137 on Rocky Linux and LTS Variants

<h2 id="introduction">Introduction</h2>
<p>This article covers a case where a job or inventory source in Ascender fails to find a collection that ships with the execution environment, because a custom <code>ANSIBLE_COLLECTIONS_PATH</code> value replaced the default collection paths instead of adding to them.</p>
<h2 id="problem">Problem</h2>
<p>Setting <code>ANSIBLE_COLLECTIONS_PATH</code> to a single project-relative path, for example <code>/opt/ansible/collections</code>, replaces the entire collections search path rather than extending it. Ansible then looks for collections only in that one directory. Collections bundled with the execution environment are no longer on the path, so their modules, plugins, and inventory plugins cannot be resolved even though they are present in the image.</p>
<h2 id="symptoms">Symptoms</h2>
<p>A task fails with a "couldn't resolve module/action" error, or an inventory source fails with an "unknown plugin" error, for a collection you know is installed in the execution environment.</p>
<p>The <code>ansible-inventory</code> or <code>ansible-config</code> diagnostic output shows a collection location that includes only your custom path, for example:</p>
<pre><code class="language-text">ansible collection location = /runner/project/collections
</code></pre>
<h2 id="resolution">Resolution</h2>
<p>Append the default collection paths to your custom value rather than replacing them. Under <strong>Settings > Jobs Settings > Extra Environment Variables</strong>, set:</p>
<pre><code class="language-json">{
  "ANSIBLE_COLLECTIONS_PATH": "/opt/ansible/collections:/runner/.ansible/collections:/usr/share/ansible/collections"
}
</code></pre>
<p>The first entry keeps your own collections directory, and the two default entries restore access to the collections bundled with the execution environment. After saving, re-run the job or inventory sync and the collection resolves correctly.</p>
<h2 id="notes">Notes</h2>
<p><code>ANSIBLE_COLLECTIONS_PATH</code> is a full replacement, not an addition. Any time you set it, include every path Ansible needs to search, not only your own. The same rule applies to <code>ANSIBLE_ROLES_PATH</code> and similar path variables.</p>
<h2 id="references--related-articles">References &#x26; related articles</h2>
<p><a href="/article/ascender/asc-custom-execution-environment">Building a Custom Execution Environment for Ascender</a><br>
<a href="/article/ascender/asc-update-execution-environment">Update the Ascender Execution Environment on Kubernetes</a><br>
<a href="https://ascender-automation.org/userguide/execution_environments.html">Ascender Execution Environments</a><br>
<a href="https://ascender-automation.org/userguide/ee_reference.html">Ascender Execution Environment Setup Reference</a><br>
<a href="https://docs.ansible.com/ansible/latest/collections_guide/collections_installing.html">Ansible collections search path</a></p>


This article covers a case where a job or inventory source in Ascender fails to find a collection that ships with the execution environment, because a custom…