
## Introduction

In environments that use user namespaces, administrators must keep user and group ID mappings consistent across multiple hosts to maintain proper access control. Many try to achieve this by symlinking `/etc/subuid` and `/etc/subgid` to shared storage, which distributes the same mappings everywhere.

However, Apptainer and similar tools rely on the `newuidmap` and `newgidmap` utilities, which refuse to read symbolic links. When you use symlinks for these files, Apptainer’s fakeroot builds fail. This article explains why this happens and shows safer ways to synchronize mappings across hosts.

## Problem

If you symlink `/etc/subuid` and `/etc/subgid` to shared storage, Apptainer’s fakeroot feature breaks. The `newuidmap` and `newgidmap` utilities reject symlinked files to prevent privilege escalation and ensure file integrity. Because these tools can’t read the mappings, any fakeroot build or container that depends on them fails.

This incompatibility prevents Apptainer from working correctly in environments that distribute user mappings through symbolic links.

## Symptoms

When you build an Apptainer image in fakeroot mode on a system with symlinked `subuid` or `subgid` files, you might see errors like:

```bash
apptainer build --fakeroot rockylinux_fakeroot.sif docker://rockylinux:9
ERROR  : 'newgidmap' execution failed. Check that 'newgidmap' is setuid root or has setcap cap_setgid+eip.
ERROR  : Error while waiting event for user namespace mappings: no event received
```

These messages show that Apptainer can't create the required user namespace mappings because `newuidmap` and `newgidmap` fail to open the symlinked configuration files.

## Root cause

`newuidmap` and `newgidmap` open the `/etc/subuid` and `/etc/subgid` files with the `O_NOFOLLOW` flag. This flag tells the kernel not to follow symbolic links. When the path ends in a symlink, the system call fails with an `ELOOP` error "Too many levels of symbolic links".

For example, an `strace` trace shows the failure:

```bash
openat(AT_FDCWD, "/etc/subgid", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW) = -1 ELOOP (Too many levels of symbolic links)
```

Because of this behavior, the utilities can't access symlinked mapping files, and Apptainer fails to set up user namespaces when using fakeroot mode.

## Resolution

To synchronize `subuid` and `subgid` files across hosts without using symlinks, consider alternative methods such as:

### `Rsync` and `Cron` jobs

Use `rsync` in conjunction with a `cron` job to periodically sync these files across hosts. For details, refer to the [Rocky Linux Rsync Guide](https://docs.rockylinux.org/guides/backup/rsync_ssh/) and [Rocky Linux Cron Guide](https://docs.rockylinux.org/guides/automation/cron_jobs_howto/).

### Ascender/Ansible automation

Automate file synchronization with Ansible for a more flexible and scalable solution. Ascender, an orchestration tool from CIQ, can assist in managing these automation tasks. For more details, refer to the [Rocky Linux Ansible Guide](https://docs.rockylinux.org/books/learning_ansible/02-advanced/) and [Ascender Automation](https://ciq.com/products/ascender/).

## References & related articles

[Rocky Linux Rsync Guide](https://docs.rockylinux.org/guides/backup/rsync_ssh/)  
[Rocky Linux Cron Guide](https://docs.rockylinux.org/guides/automation/cron_jobs_howto/)  
[Rocky Linux Ansible Guide](https://docs.rockylinux.org/books/learning_ansible/02-advanced/)  
[Ascender Pro](https://ciq.com/products/ascender/)


Error When Symlinking User and Group ID Mapping


## Introduction

This document outlines how to add an SSL certificate to an existing Ascender installation, assuming you have a valid and working Ascender installation. The recommended installation method is the [ascender-install](https://github.com/ctrliq/ascender-install) script from the CIQ repository.

A requirement for successfully adding an SSL certificate is to have one initially generated. This is not discussed in this article and creating an SSL certificate is necessary before continuing with this guide. The Common Name (CN) field in the certificate needs to exactly correlate with the URL you have set for your Ascender instance. Both the certificate and private key must be in `PEM` format. The certificate has to be stored with a `.crt` or `.cer` file extension and the private key assigned a `.key` file extension.

## Problem

When Ascender was installed, the `http` option was chosen instead of `https`. As a result, Ascender is using a default self-signed certificate and has no options within the `custom.config.yml` file to use other certificates.

## Resolution

After installing Ascender with the [ascender-install](https://github.com/ctrliq/ascender-install) scripts from CIQ, a `custom.config.yml` file is created. This includes all the variables required for the initial install process. Change each of the variables in this file like the example below:

* **kube_install**: In your case, a kubernetes cluster on Ascender is running already. This value needs to be set to `false`.
* **download_kubeconfig**: Similarly to the above, this value also needs to be configured as `false`. This is because a valid `KUBECONFIG` file used for authentication already exists in your Ascender instance.
* **k8s_lb_protocol**: Change this from `http` to `https`.

Go to the bottom of the file and include these lines:

* **tls_crt_path**: Details the path of where the certificate is located.
* **tls_key_path**: Here will be the path to where the private key is found.

An example of the above configuration steps:

```yaml
tls_crt_path: ~/ascender.crt
tls_key_path: ~/ascender.key
```

Run the `setup.sh` script again and you will find the new certificate successfully installed. Run the command illustrated below to observe if the new certificate has been successfully applied or not:

```bash
echo | openssl s_client -connect your-ascender-hostname:443 2>/dev/null | openssl x509 -noout -subject -ext subjectAltName -dates
```

## References & related articles

[CIQ Ascender](https://ciq.com/products/ascender)  
[Ascender Install Scripts and Documentation](https://github.com/ctrliq/ascender-install)  
[Ascender Feature Overview](https://ciq.com/blog/manage-enterprise-infrastructure-with-ascender-by-ciq-what-is-ascender/)  
[Ascender Installation Video](https://www.youtube.com/watch?v=j15pFNfs8dw)  
[Generating SSL Keys](https://docs.rockylinux.org/guides/security/ssl_keys_https/)  
[Generating SSL Keys with Let's Encrypt](https://docs.rockylinux.org/guides/security/generating_ssl_keys_lets_encrypt/)  


Add an SSL Certificate to an Ascender Instance That Was Set Up Without a Certificate


## Introduction

This guide will have you setup and run Ascender on a K3s cluster.

## Problem

Many customers face difficulties when managing their infrastructure at scale. This is where Ascender comes in to ensure scalable patching, security hardening and more across your fleet.

## Symptoms

A new user may encounter difficulties installing Ascender onto their K3s cluster.

## Resolution

### Prerequisites

* Same OS family as Rocky Linux. Rocky Linux 8.x or Rocky Linux 9.x is recommended as the base OS.

* System requirements:

  * CPUs: 2

  * Memory: 8GB (if installing both Ascender and Ledger)

  * Disk: 20GB (if installing both Ascender and Ledger)

* Internet access for the K3s cluster.

* Perform the below steps with a user that has privilege escalation. Do not use the `root` user account.

### Setup

* Update the node running K3s:

```bash
sudo dnf update -y
```

* Install `git`:

```bash
sudo dnf install -y git
```

* Allow these ports in `firewalld`:

```bash
sudo firewall-cmd --permanent --zone=public --add-port=22/tcp
sudo firewall-cmd --permanent --zone=public --add-port=80/tcp
sudo firewall-cmd --permanent --zone=public --add-port=443/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5432/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5895/tcp
sudo firewall-cmd --permanent --zone=public --add-port=5986/tcp
sudo firewall-cmd --reload
```

* Clone the Ascender repository:

```bash
git clone https://github.com/ctrliq/ascender-install.git
```

* Navigate to the installation directory:

```bash
cd ascender-install
```

* Run the `config_vars.sh` script:

```bash
./config_vars.sh
```

* The script will present you with multiple questions. Answer each of them according to how you want to set up your Ascender environment and if you already have K3s installed on your node.

* Run the setup script to install Ascender:

```bash
sudo ./setup.sh
```

* Once complete, the final playbook recap will look like the following example:

```bash
PLAY RECAP *************************************************************************************************************************
ascender_host              : ok=14   changed=6    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0
localhost                  : ok=72   changed=27   unreachable=0    failed=0    skipped=4    rescued=0    ignored=0

ASCENDER SUCCESSFULLY SETUP
```

### Accessing Ascender

* You can access Ascender directly via the internal CLUSTER IP address.

## Notes

* To monitor the Ascender setup on your K3s cluster during install:

```bash
kubectl get pod -n ascender -w
```

* For further details surrounding particular pods:

```bash
kubectl describe pod -n ascender [pod name]
```

## References & related articles

[Ascender General Prerequisites](https://github.com/ctrliq/ascender-install/tree/main?tab=readme-ov-file#general-prerequisites)
[Ascender K3s Install Guide](https://github.com/ctrliq/ascender-install/blob/main/docs/installation/k3s/README.md)


Quick Start Guide for Installing Ascender on K3s


## Introduction

When importing jobs or running playbooks in Ascender, users may encounter a warning message indicating a version mismatch between the AWX Galaxy Collection and the Ascender version.

## Problem

During job operations in Ascender, the following warning message is displayed:

```text
[WARNING]: You are running collection version 24.6.1 but connecting to AWX version 25.2.0.
```

This warning appears when the `awx.awx` Galaxy collection version is lower than the Ascender application version.

## Resolution

This warning message can be safely ignored and does not require any action. The warning indicates that the `awx.awx` Galaxy collection is pinned to a specific version (in this example, `24.6.1`), while the Ascender application itself is running a newer version (in this example, `25.2.0`). `24.6.1` is the latest release available for the `awx.awx` galaxy collection.

No action is required. The job import and execution will proceed normally despite this warning. The warning is informational only and does not indicate an error or compatibility issue.

## References & related articles

[Ascender Quick Start Guide](https://kb.ciq.com/article/ascender/asc-ascender-quick-start)
[Ascender General Prerequisites](https://github.com/ctrliq/ascender-install/tree/main?tab=readme-ov-file#general-prerequisites)


AWX Collection Version Warning During Job Run


## Introduction

This document outlines how to update the SSL certificate for an existing Ascender installation. This guide will provide you with two options to update the certificate. It assumes that you have a valid and working Ascender installation. The recommended method for installation is the [ascender-install](https://github.com/ctrliq/ascender-install) script from the CIQ repository. The first certificate update method uses the Ascender installation scripts.

Obtaining SSL certificates for your environment is outside the scope of this article, and you will need to have them issued before proceeding. The Common Name (CN) field in the certificate must match the URL you have chosen for Ascender. Your certificate and private key must be in PEM format, with the certificate stored in a `.crt` (or `.cer`) file and the private key stored in a separate `.key` file.

## Problem

Depending on your certificate provider, you will need to update the SSL certificate for your Ascender install every 90 days to 1 year if you are using a domain validated certificate.

## Resolution

When updating your certificate, you can choose to rerun the setup script or manually update the secret within Kubernetes.

### Rerun the setup script

When you install Ascender using the [ascender-install](https://github.com/ctrliq/ascender-install) scripts provided by CIQ, a `custom.config.yml` file is generated that contains the variables used to perform your initial install. Modify the following variables within this file:

* **kube_install**: Configure this as `false`, as you already have a Kubernetes cluster on which Ascender is running.
* **download_kubeconfig**: Modify this to `false`, as you already have a valid `KUBECONFIG` file to authenticate to your existing Ascender instance.
* **tls_crt_path**: Change this, if necessary, to the location of your certificate.
* **tls_key_path**: Edit this if required, to the place where you stored your private key.

Now, rerun the `setup.sh` script and the new certificate will be installed.

### Modify the Kubernetes secret

To manually deploy the certificate within Kubernetes, you need to create a new secret manifest. Specifically, we will modify the `ascender-tls-secret` file. First, we need to `base64` encode our certificate and private key:

```bash
ASCENDER_CERT=$(base64 -w0 /path/to/crt)
ASCENDER_KEY=$(base64 -w0 /path/to/key)
```

Next, we will create a new manifest using this information:

```bash
cat <<EOF > ascender_tls.yml
apiVersion: v1
kind: Secret
metadata:
  name: ascender-tls-secret
  namespace: ascender
type: kubernetes.io/tls
data:
  tls.crt: ${ASCENDER_CERT}
  tls.key: ${ASCENDER_KEY}
EOF
```

You can apply this by running the following:

```bash
kubectl apply -f ascender_tls.yml
```

After a few seconds, your new certificate will be applied. You can verify your new certificate, by running the following:

```bash
echo | openssl s_client -connect your-ascender-hostname:443 2>/dev/null | openssl x509 -noout -subject -ext subjectAltName -dates
```

## References & related articles

[CIQ Ascender](https://ciq.com/products/ascender-pro)  
[Ascender Install Scripts and Documentation](https://github.com/ctrliq/ascender-install)  
[Ascender Feature Overview](https://ciq.com/blog/manage-enterprise-infrastructure-with-ascender-by-ciq-what-is-ascender/)  
[Ascender Installation Video](https://www.youtube.com/watch?v=j15pFNfs8dw)


Update SSL Certificates for Ascender


## Introduction

Fuzzball Orchestrate is architected as a collection of microservices deployed in a Kubernetes (K8s)
cluster. To understand what is happening with the Fuzzball deployment, you must be able to inspect
the status of the pods supporting these containerized microservices, and gather relevant logs.

## Checking pod status

Any Fuzzball troubleshooting starts with confirming the presence and inspecting the status of
containers/microservices within their respective pods.

You can view and inspect the status of pods hosting containerized microservices using the standard
K8s control plane tool, `kubectl`. After executing the following `kubectl` command, a table
is displayed giving information about all of the pods within the K8s cluster.

```text
# kubectl get pods -A
NAMESPACE            NAME                                                       READY   STATUS      RESTARTS      AGE
cert-manager         cert-manager-56f459fcb7-j7wmt                              1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-56f459fcb7-m8gq2                              1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-56f459fcb7-qj9zd                              1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-cainjector-78f497d9d8-5fc5r                   1/1     Running     1 (35m ago)   19d
cert-manager         cert-manager-webhook-5bf5cccd59-qg5qs                      1/1     Running     1 (35m ago)   19d
cert-manager         trust-manager-778f7b488-gdmrh                              1/1     Running     1 (35m ago)   19d
database             database-debug-0                                           1/1     Running     1 (35m ago)   19d
database             database-postgresql-0                                      1/1     Running     1 (35m ago)   19d
fuzzball-system      fuzzball-operator-controller-manager-55cc86985f-c7tzn      2/2     Running     2 (35m ago)   19d
fuzzball             audit-archive-28836720-4wcf5                               0/1     Completed   0             2d6h
fuzzball             audit-archive-28838160-86zjk                               0/1     Completed   0             30h
fuzzball             audit-archive-28839600-tbhj2                               0/1     Completed   0             6h25m
fuzzball             fuzzball-account-7749678b7c-m272g                          1/1     Running     1 (35m ago)   19d
fuzzball             fuzzball-account-external-5f5cd6f56-86k4z                  1/1     Running     1 (35m ago)   19d
fuzzball             fuzzball-account-worker-7f69c98747-gbtvc                   1/1     Running     4 (33m ago)   19d
fuzzball             fuzzball-admin-0                                           1/1     Running     1 (35m ago)   19d
[...snip]
```

If you want to view only pods running within the `fuzzball` namespace, you can change the command
like so:

```text
# kubectl get pods -n fuzzball 
NAME                                             READY   STATUS      RESTARTS       AGE
audit-archive-28836720-4wcf5                     0/1     Completed   0              2d8h
audit-archive-28838160-86zjk                     0/1     Completed   0              32h
audit-archive-28839600-tbhj2                     0/1     Completed   0              8h
fuzzball-account-77bc99544-n2dgb                 1/1     Running     0              119s
fuzzball-account-external-7977f8c5dc-hmfhf       1/1     Running     0              119s
fuzzball-account-worker-6b646c8b76-jq2sf         1/1     Running     0              119s
fuzzball-admin-0                                 1/1     Running     1 (173m ago)   19d
fuzzball-audit-74bb656b6b-lltx4                  1/1     Running     0              119s
fuzzball-auth-spicedb-57bb79cf9b-k2zh7           1/1     Running     0              119s
fuzzball-dns-569f675947-cf59z                    1/1     Running     0              119s
fuzzball-kube-68fc8d8b8d-f7gkg                   1/1     Running     0              119s
fuzzball-log-845fc88dd9-bvr4c                    1/1     Running     0              119s
fuzzball-openapi-686488649-7fj2s                 1/1     Running     0              119s
fuzzball-organization-7f7f5f6878-qs55p           1/1     Running     0              119s
fuzzball-organization-external-cc78884d5-z56j5   1/1     Running     0              118s
fuzzball-organization-worker-55599cd8d8-htknd    1/1     Running     0              118s
fuzzball-permission-external-55d965c957-n5fsv    1/1     Running     0              118s
fuzzball-schedule-0                              1/1     Running     0              170m
[...snip]
```

Note the values in the `READY` column indicating `1/1`, (or potentially `2/2`, etc.) and the
`STATUS` column indicating `Running`. Any pods that do not report a `Running` or `Completed` status,
or those that report fewer instances than expected should be investigated further by inspecting
logs.

## Retrieving and Inspecting logs

While provisioning Fuzzball within a K8s cluster (using `kubectl apply -f fuzzball.yaml`), you can
watch the log output with the following command:

```text
# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1
```

You can stream logs from an individual pod using the following syntax. Note that the namespace must
be specified since Fuzzball pods are not deployed in the "default" namespace. For example, to view
the logs streaming from the `fuzzball-schedule-0` pod, you could use the following command. (Omit
the `--follow --tail=0` options/arguments if you want to view the log at a single instant without
streaming it.)

```text
# kubectl logs -n fuzzball fuzzball-admin-0 --follow --tail=0
```

The `kubectl logs` command outputs verbose logs in json format. These can be filtered and
parsed into human-readable output using `jq` like so:

```text
# kubectl logs -n fuzzball fuzzball-admin-0 | \
    jq -r 'select(.level != "debug") | [.ts, .msg, .Error] | @tsv'
```

The `kubectl` utility does not provide a straightforward command for capturing all Fuzzball logs in
a single stream. However, the logs for a collected Fuzzball “app” can be inspected using a label
selector like so. (This is the strategy used above to stream logs during the provisioning process.)

```text
# kubectl logs --prefix --namespace=fuzzball --all-containers \
    --max-log-requests=99 --tail=0 --follow \
    --selector='app.kubernetes.io/name=fuzzball-schedule'
```

The list of available labels can be viewed using `kubectl get pods -n fuzzball --show-labels`.

Finally, it may be useful to gather a full log dump (for example, to send to CIQ support). This can
be accomplished by requesting logs from all app labels. The app labels may change with different
versions of Fuzzball and can be inspected as described above.  Here is an example:

```text
# kubectl logs --prefix --all-containers --max-log-requests=99 \
    --namespace=fuzzball \
    --selector='app.kubernetes.io/name in (fuzzball-account, fuzzball-admin, fuzzball-audit, fuzzball-auth, fuzzball-dns, fuzzball-kube, fuzzball-log, fuzzball-openapi, fuzzball-organization, fuzzball-permission, fuzzball-schedule, fuzzball-secret, fuzzball-storage, fuzzball-sync, fuzzball-ui, fuzzball-user, fuzzball-workflow)'
```

## Restarting Fuzzball

You can invoke the command `kubectl rollout restart` to restart the Fuzzball deployment. For
example:

```text
# kubectl rollout restart deployment -n fuzzball
```

Other namespaces that may be pertinent for restart include

- `database` - provides the PostgreSQL database that backs Fuzzball Orchestrate state
- `ingress` - maps HTTP(S) URLs to relevant Fuzzball services, including the Fuzzball UI, Fuzzball API, and Keycloak
- `kyverno` - provides a policy engine used by Fuzzball Orchestrate
- `workflow` - provides the temporal service that is used by Fuzzball Orchestrate
- `identity` - provides a Keycloak instance for identity and access management

## Uninstalling Fuzzball

If a deployment failed or you have a need to uninstall a deployed Fuzzball installation, you may
need to tear down the installation and start fresh. The following command will delete an existing
Fuzzball deployment, allowing you to start over if needed.

```text
# kubectl delete FuzzballOrchestrate fuzzball-orchestrate
```

You can watch the deprovisioning process as it occurs with the following command:

```text
# kubectl logs -n fuzzball-system -l app.kubernetes.io/name=fuzzball-operator -f --tail=-1
```

You may also want to remove the `substrate` configuration directory that is shared out to compute
nodes via NFS.  For instance:

```text
# rm -rf /srv/fuzzball/shared/substrate
```

This directory will be recreated if you re-provision your Fuzzball instance.


Basic Troubleshooting Procedures


## Introduction

The Fuzzball Web GUI allows you to view logs from your running or completed job.

## Problem

Under some circumstances, a user may click on the "Logs" button within the job section of the
workflow dashboard and receive the message "No log output" even though the job completed
successfully and logs were expected. The problem may be that the user does not have the appropriate
permissions to view the logs, or that the browser cannot access the API endpoint because Fuzzball
was installed using self-signed certificates.

## Symptoms

Here is an example showing the problem. The user has clicked on the "Logs" button for a workflow
that was expected to produce logs and is seeing the message "No log output".

![No log output in GUI](/images/articles/no-logs.png)

## Resolution

### Ensure you have permission to view logs

First, ensure that you have the appropriate permissions to view logs in the workflow.  If you are
viewing a job in a workflow submitted by another member of your shared account, you might need to
ask them for permission to view the logs.  They can add you as a job owner using the following CLI
command:

```text
fuzzball workflow add-owner <workflow UUID> <user UUID>
```

The workflow UUID and user UUID can be obtained with the `fuzzball workflow list` and `fuzzball
account list-members` commands respectively.  After you have been added as an owner you should be
able to view logs from jobs within the workflow.

### Self-signed certificates require an exception

If you have the correct permission to view job logs, you should check to make sure that your browser
can access them from the API.  When self-signed certificates are used, the Fuzzball GUI asynchronous
access to the Fuzzball Orchestrate API can be blocked by your web browser's security policy, causing
an erroneous indication that no logs are available.

To correct this issue, you can direct your browser to the Fuzzball API endpoint.  The exact URI will
depend on the way that Fuzzball was installed, but it may be accessible by changing the string `ui`
in the Fuzzball URL to `api`.  For instance the following refers to the URL for the Fuzzball web GUI
and the API on an example installation respectively.

- <https://ui.stable.fuzzball.ciq.dev/>
- <https://api.stable.fuzzball.ciq.dev/>

In the case of a self-signed certificate, your browser may complain that the URL is unsafe and may
prompt you to create an exception.  After doing so, a browser refresh should allow you to view the
logs in the web GUI.


Troubleshooting Missing Job Logs in Fuzzball


## Introduction

If no certificates are supplied by the Administrator during the configuration and installation
procedure, the Fuzzball Operator will create self-signed certificates for hosting Fuzzball
Orchestrate.  

## Problem

You may run into issues using the Fuzzball CLI if the cluster (context) you are accessing is using
a self-signed certificate.

## Symptoms

If you receive an error like the following when attempting to use the Fuzzball CLI it suggests that
your environment is not properly configured to recognize the self-signed TLS certificates in use.

```text
tls: failed to verify certificate: x509: certificate signed by unknown authority
```

## Resolution

An administrator can use the following commands to export the relevant certificates from the
Fuzzball K8s installation:

```text
# mkdir certs

# kubectl get secret -n cert-manager root-ca-cert -o "jsonpath={.data['ca\.crt']}" | base64 --decode >certs/ca.crt

# kubectl get secret -n cert-manager root-ca-cert -o "jsonpath={.data['tls\.crt']}" | base64 --decode >certs/tls.crt
```

These certificates can be distributed to the systems where the Fuzzball CLI is installed and then
used by adding an environment variable like so:

```text
$ export SSL_CERT_DIR=/path/to/certs
```

Alternately, CLI users can set an environment variable, `FUZZBALL_INSECURE=true` or use the
`--insecure` flag when invoking the Fuzzball command.

## Notes

The solutions selected here are appropriate for development clusters, testing, and debugging, but it
is ultimately more secure for administrators to use certificates issued by 3rd parties.  The
Fuzzball documentation contains a section describing how to do this.


Using the CLI with Self-Signed TLS Certificates


## Introduction

CIQ provides [Long Term Support](https://ciq.com/services/long-term-support/) for Rocky Linux minor releases that have reached End of Life. This article explains how CIQ maintains and updates packages in the LTS repository.

## Standard repositories vs. LTS repository

### Standard repositories

When a Rocky Linux minor release reaches End of Life, for example Rocky 9.2, CIQ freezes all standard repositories in their final state and makes them available in Portal:

- Rocky Linux 9.2 BaseOS - `rocky-9.2-baseos.x86_64`
- Rocky Linux 9.2 AppStream - `rocky-9.2-appstream.x86_64`
- Rocky Linux 9.2 extras - `rocky-9.2-extras.x86_64`

These repositories and others aren't updated after End of Life. They contain the original package versions from that release to satisfy package installation and dependencies.

### LTS repository

CIQ maintains a separate LTS repository where patched package updates are published, for example:

- Rocky Linux 9.2 CIQ Long Term Support - `rlc-9.2-lts.x86_64`

When CIQ patches packages for CVEs or critical bugs, CIQ publishes the updated packages to the LTS repository.

## How to Use LTS Repositories

To find patched packages, use the LTS repository instead of standard repositories when using an LTS product.

### Locating patched packages

In the CIQ customer portal, navigate to the LTS repository for your Rocky Linux version:

```text
https://portal.ciq.com/products/88eb2f09-3146-49c3-b0ef-d2ea9705258e/repository/rlc-9.2-lts.x86_64
```

Browse to the appropriate package directory, for example `/Packages/s/` for packages starting with 's'.

### Installing from LTS

If you configure the LTS repository on your system, patched packages automatically update during regular updates because they have higher version numbers than the frozen standard repositories:

```bash
dnf update
```

## Notes

- LTS repositories are actively maintained throughout the support lifecycle.
- Standard repositories remain frozen for reference and reproducibility.
- Security-patched versions referenced in CIQ CSAF advisories are always in the LTS repository.
- Patched packages in LTS have higher version numbers and automatically upgrade over frozen packages during `dnf update`.

## References & related articles

[CIQ Customer Portal](https://portal.ciq.com)
[Rocky Linux Documentation](https://docs.rockylinux.org)
[Rocky Linux Long Term Support](https://ciq.com/services/long-term-support/)


Understanding CIQ Long Term Support Repositories


## Introduction

The need for secure and efficient file transfer solutions has become paramount. Whether it is sharing sensitive data, collaborating on projects, or automating workflows, a reliable Secure File Transfer Protocol (SFTP) server can streamline these processes, while ensuring data integrity and confidentiality.

The CIQ Secure File Exchange (SFE) service is an SFTP, FTP/S, HTTP/S, and WebDAV server. It provides a flexible foundation for managing file transfers while offering an efficient and scalable solution for storing and retrieving data. The service supports private virtual folders per user, which can also be shared with other users as well.

## How to access the service

To have an SFE account generated, please create a ticket via the CIQ Support Portal. A member of support will then generate your account and securely send you the login credentials via the ticket.

Login to the [CIQ SFE Client Portal](https://sftp.ciq.com/web/client/login) using your provided credentials.

You will then be prompted to change your password. Please do so and store the new password in a secure location.

## How to upload files

Files can be uploaded in two ways: via your web browser or the CLI.

### Web browser method

Log in with your credentials at the [CIQ SFE Client Portal](https://sftp.ciq.com/web/client/login).

Navigate to the `My Files` section and press the `Upload files` button.

![The CIQ SFE Client's My Files section and highlighting the "Upload" button that can be clicked near the top left-hand corner of the screen, just below the word "show".](/images/articles/ciq_sfe_file_upload_1.webp)

Select your files and then hit `Submit`:

![Highlighting the area where the user can upload their files and then clicking the "Submit" button to start the upload.](/images/articles/ciq_sfe_file_upload_2.webp)

The file will be uploaded to your virtual folder. Please inform the Support Team via your support ticket and they can retrieve the files and start their investigation.

**If you would like to further customize how you share your files, please follow the steps below:**

Under `My Files`, click the checkboxes next to the files you uploaded and then click the `Share` button:

![The "My Files" section, with the files that the user wants to share having their checkboxes ticked on the left-hand side of the screen, in the same column as the "Upload" button.](/images/articles/ciq_sfe_file_upload_4.webp)

![The "My Files", highlighting the "Share" button to click that is on the left of the refresh button of the top of the screen.](/images/articles/ciq_sfe_file_upload_3.webp)

At the next screen, add a `Name` for your share, set a `Password` and `Expiration` date if you desire and then select `Submit`:

![The "Add a new share" section of the CIQ SFE Client, showing a "Name" can be added for the share near the top of the page, the "Scope" can be changed from "Read" to "Write" to "Read and Write" under the "Name" section, a Password that can be added around the middle of the page, as well as setting an Expiration date just below the Password entry field. Finally the "Submit" button is highlighted in green for the user to click and the share can be created.](/images/articles/ciq_sfe_file_upload_5.webp)

Then in the `Shares` screen, select the share that you created and click the `Link` button at the top of the page:

![The test share that was created in the previous step and which has been selected for sharing by the user.](/images/articles/ciq_sfe_file_upload_7.webp)

![The "Shares" section of the CIQ SFE Client, showing the "Link" button in the top left-hand corner under the word "entries", where links to the share can be generated.](/images/articles/ciq_sfe_file_upload_6.webp)

Finally, copy the link type you would like to share and include the link in the relevant ticket. The Support Team can then easily access your files:

![Once the "Link" button is clicked, three links are generated: 1) The user can download the share as a zip file. 2) The user can access a directory to view all of the files. 3) If only a single file is shared, the user can download the file uncompressed.](/images/articles/ciq_sfe_file_upload_8.webp)

### CLI method

Run the following command to access the CIQ SFE service via the CLI. Your username is usually your email address:

```bash
sftp -P 2022 <YOUR_USERNAME>@sftp.ciq.com
```

Enter your password when prompted.

Create a directory where you would like to store your files:

```bash
sftp> mkdir <YOUR_DIRECTORY>
```

Upload your files using the `put` command:

```bash
put ./text.txt ./test/
```

Please check the example below for uploading a file called `test.txt` to a directory in the CIQ SFE service called `test`:

```bash
sftp> put ./test.txt test/
Uploading ./test.txt to /test/test.txt
test.txt                                                                                                                                      100%   13     0.1KB/s   00:00
sftp>
```

Once uploaded, please check that the files were uploaded successfully. An example is shown below:

```bash
sftp> ls -l test/
-rwxr-xr-x    1 991      991            13 Jan 20 02:32 test.txt
```

Please inform the Support Team via your ticket that the files have been uploaded successfully.


**_ℹ️ NOTE_** _For all methods listed above, **please provide a sha256sum in the ticket for each file that is in the multiple gigabytes**. This allows the Support Team to easily verify if an upload has been successful or not and can continue their investigation without further interruptions to the customer._

If you run into any problems whilst uploading files to the CIQ SFE service, please don't hesitate to make the Support Team aware and they will assist you with the upload process.


How to Upload Files to the CIQ Secure File Exchange (SFE) Service


## Introduction

When you need help, the CIQ Portal is your starting point for creating a support request. Submitting a ticket there ensures your request reaches our Support Team, who will review, respond, and work to resolve the issue as quickly as possible in line with the agreed [Service Level Agreements (SLAs)](https://ciq.com/legal/sla). Please note that all support requests are managed in accordance with [CIQ's terms and conditions](https://ciq.com/legal/terms-conditions/).

## How to create a ticket

### Ticket creation via CIQ Portal

This is the primary way to generate a ticket and make us aware of your issue. Log into the CIQ Portal at [https://portal.ciq.com](https://portal.ciq.com)

**_ℹ️ NOTE:_** _If you are unable to log in to the Portal for any reason, please contact your CSM and they will assist you with access._

Once logged in, click the `Support` button on the left-hand menu. You will see a grid of your existing tickets. To create a new ticket, click the green `+ Create Support Request` button.

Provide the following information:

* `Subject` - a brief description of the problem.

* `Product` - the product you are having trouble with. The available options include Rocky Linux, Warewulf Pro, Warewulf, RLC, and more.

* `Version` - select the version of the product you are experiencing the issue with from the drop-down menu.

* `Priority` - this ranges from `Urgent` to `Low`. Please consult CIQ's [terms and conditions](https://ciq.com/legal/terms-conditions/) before selecting the `Priority`, to ensure that CIQ can provide you with the appropriate level of support.

* `Add collaborators` - optionally choose who to include from your organization on the ticket.

* `Description` - provide as much detail as possible surrounding the issue.
    **_ℹ️ NOTE:_** _If you can [generate an sosreport](https://kb.ciq.com/article/rocky-linux/rl-generate-an-sosreport) or attach additional outputs to help describe the problem to the ticket, that would greatly speed up the resolution process and avoid the Support Team having to ask for additional information._

* `Attachments` - files of up to `50MB` in size can be attached directly to the ticket. If your file type is not supported, please compress it into a `.zip`, `.tar`, or `.gz` file. For files larger than `50MB`, please visit [sftp.ciq.com](https://sftp.ciq.com) for upload. Please ask the Support Team to create a Secure File Exchange account for you.

Once all of the above information is filled out, click the `Submit` button.

You will then be taken to your ticket. All communication with the CIQ Support Team is done here until successful resolution of the issue.

To view all of your tickets, click the `Support` button on the left-hand menu in the CIQ Portal. Under the `Requested` tab, you can see all of the Support Tickets you have created. You can also see all of the tickets from your team members under the `Organization` tab, as well as tickets you have been CC'd in.

### Ticket creation via email

You can also generate tickets by sending an email directly to [support@ciq.com](mailto:support@ciq.com)

The `Subject Line` contains the same information as the `Subject` field when creating a Support Ticket.

The body of the email is the same as the `Description` field listed above in the support ticket creation section.

**_ℹ️ NOTE:_** _Any tickets created by this method are automatically placed as `Normal` (`Severity 3`) `Priority` tickets. If you have an `Urgent` issue, please let the CIQ Support Team know via the ticket and they will raise the `Priority` as needed._

You can also respond to replies from the CIQ Support Team via email as well.

## References & related articles

[Generating sosreports on Rocky Linux](https://kb.ciq.com/article/rocky-linux/rl-generate-an-sosreport)
[How to Upload Files to the CIQ Secure File Exchange (SFE) Service](https://kb.ciq.com/article/general/ciq-secure-file-exchange)
[What Data is Collected in an sosreport on Rocky Linux](https://kb.ciq.com/article/rocky-linux/rl-what-information-an-sosreport-collects)


How to Create a Support Ticket


## Introduction

This article demonstrates how to safely add additional swap space to a Rocky Linux system while it's running and actively using existing swap. This procedure is non-disruptive and doesn't require a system reboot or affect running processes.

## Problem

A system administrator needs to increase available swap space on a server that's currently using its existing swap partition. The system has 2 GB of swap space configured as an LVM partition and requires an additional 10 GB of swap capacity without interrupting any active processes or services.

## Resolution

### Create a Swap File

Instead of modifying existing partition sizes, create a new swap file on the root filesystem. This approach avoids the complexity of resizing LVM volumes and can be done while the system is running.

#### Create the swap file using fallocate

```bash
sudo fallocate -l 10G /swapfile_10gb
```

#### Set appropriate permissions

```bash
sudo chmod 600 /swapfile_10gb
```

#### Format the file as swap

```bash
sudo mkswap /swapfile_10gb
```

#### Enable the new swap file

```bash
sudo swapon /swapfile_10gb
```

This command activates the swap file immediately without affecting your existing swap partition or any running processes.

#### Verify both swap spaces are active

```bash
sudo swapon --show
```

Expected output should show:

- Your original swap partition (e.g., `/dev/mapper/rl-swap` - 2.0GB)
- Your new swap file (`/swapfile_10gb` - 10GB)
- Total: 12 GB of swap space

You can also verify with:

```bash
free -h
```

### Make the Change Persistent

To ensure the system activates the swap file automatically after reboots, add an entry to `/etc/fstab`:

```bash
echo '/swapfile_10gb none swap defaults 0 0' | sudo tee -a /etc/fstab
```

#### Reload systemd configuration

```bash
sudo systemctl daemon-reload
```

## Notes

### Swap Files vs Swap Partitions

Both swap files and swap partitions provide equivalent performance on modern filesystems. Swap files offer several advantages:

- No need to modify partition tables or LVM volumes
- You can create, resize, or remove them without downtime
- Multiple swap files can coexist with swap partitions
- Easier to manage on systems with complex storage configurations

### Filesystem Compatibility

Rocky Linux typically uses XFS for the root filesystem, and both XFS and ext4 fully support non-sparse swap files created with `fallocate`. Modern kernels treat a properly allocated swap file the same as a swap partition for reliability and performance. Using `fallocate` offers advantages over older `dd if=/dev/zero` examples because it's faster and avoids writing unnecessary zeroes.

Note: Advanced or copy-on-write filesystems, for example Btrfs, can require extra steps such as setting the "no copy-on-write" attribute before creating a swap file. This isn't a concern for default Rocky Linux XFS or ext4 deployments and is outside the scope of this article.

### Swap Priority

By default, the kernel uses swap spaces with equal priority. If you need to prioritize one swap space over another, you can specify the `pri` option in `/etc/fstab`, for example:

```bash
/swapfile_10gb none swap defaults,pri=10 0 0
```

The kernel uses higher priority values first. Priority values range from -2 (lowest, the default) to 32767 (highest). By default, if no priority is specified, the kernel assigns a priority of -2.

## References & related articles

[Rocky Linux Documentation Filesystem - swapon swapoff](https://docs.rockylinux.org/labs/systems_administration_II/lab6-the_file_system/?h=swap#swapon-and-swapoff)
[How to Limit Tmpfs for Taking Swap Space](https://kb.ciq.com/article/warewulf/ww-tmpfs-swap-memory)


Adding Swap Space to a Live Rocky Linux System


## Introduction

At CIQ, customers will often create tickets due to a kernel crash they experienced. As part of the investigation process, the Support Team asks for a `vmcore` dump to be uploaded to CIQ's Secure File Exchange (SFE) service for analysis.

For more information on how to upload a `vmcore` dump or other files to the SFE service, please consult the [How to Upload Files to the CIQ Secure File Exchange (SFE) Service](https://kb.ciq.com/article/general/ciq-secure-file-exchange) KB article.

This article will detail some of the steps that can be taken with the Crash Utility for `vmcore` dump analysis.

For further reading, the author highly recommends the following resources:

* [Crash Utility White Paper by David Anderson](https://crash-utility.github.io/crash_whitepaper.html)

* [Linux Kernel Crash Book - Everything you need to know by Igor Ljubuncic](https://www.dedoimedo.com/computers/www.dedoimedo.com-crash-book.pdf)

## Example issue

A customer created a ticket with CIQ, after their CentOS 7.9 node using CIQ Bridge had a kernel panic and rebooted. What follows will be how to setup the Crash Utility and the steps taken to debug the issue.

### Crash Utility environment setup

* Set up a Rocky Linux 9.x installation on a node size of your choice.

* Update all packages:

```bash
sudo dnf update -y
```

* Install the `kexec-tools` and `crash` packages:

```bash
sudo dnf install -y kexec-tools crash
```

* From the node that went into a kernel panic state, find the kernel version:

```bash
uname -r
```

* With the kernel version identified, search and download the equivalent version of the `kernel-debuginfo-common` package. In the CentOS 7.9 example listed above, the kernel version was `3.10.0-1160.119.1.el7.x86_64`, thus the required package was `kernel-debuginfo-common-x86_64-3.10.0-1160.119.1.el7.x86_64.rpm`.

* Similarly, the same version of the `kernel-debuginfo` package is also needed. Again referencing the CentOS 7.9 example, downloading the `kernel-debuginfo-3.10.0-1160.119.1.el7.x86_64.rpm` package is required.

* **Note for Rocky Linux users** - to find the appropriate `kernel-debuginfo-common` and `kernel-debuginfo` packages, go to [The Rocky Linux Vault](https://dl.rockylinux.org/vault/rocky/). Using Rocky Linux 9.5 with the x86_64 architecture as an example, the `kernel-debuginfo-common` package is found [in the devel repository](https://dl.rockylinux.org/vault/rocky/9.5/devel/x86_64/os/Packages/k/) and `kernel-debuginfo` [is in the devel repository under debug](https://dl.rockylinux.org/vault/rocky/9.5/devel/x86_64/debug/tree/Packages/k/).

* Install both `kernel-debuginfo-common` and `kernel-debuginfo` packages using the `localinstall` command. The order the packages are installed in also matters:

```bash
sudo dnf install -y <path_to_kernel-debuginfo-common_rpm>
sudo dnf install -y <path_to_kernel-debuginfo_rpm>
```

* **Note for CentOS users** - the `vmcore` dump will be generated as a `vmcore.flat` file. This is not natively readable by the Crash Utility and thus has to be converted into a `vmcore` dump file. Perform the following command to convert the file:

```bash
/usr/sbin/makedumpfile -R ./vmcore < ./vmcore.flat
```

* Finally, start the Crash Utility by setting the `vmlinux` version as an argument under your `/usr/lib/debug/lib/modules/<kernel_version>` directory and pointing to the `vmcore` dump file:

```bash
crash /usr/lib/debug/lib/modules/<kernel_version>/vmlinux <path_to_vmcore_dump>
```

### Example crash dump analysis

* Starting the Crash Utility will bring up an overview of the `vmcore` dump, including the date the dump was taken, node name, memory availability, kernel version, and the kernel panic reason.

* An example kernel panic that CIQ often sees from customers running Kubernetes is `blocked tasks`:

```bash
"Kernel panic - not syncing: hung_task: blocked tasks"
```

* From the initial overview, the amount of memory available was plentiful during the kernel panic. However, as you will see later on in the article, having plentiful memory does not rule out a memory-related issue:

```bash
MEMORY: 1011.7 GB
```

* For deeper analysis on memory usage and availability, the `kmem -i` command can be run and produces an output similar to the below example:

```bash
crash> kmem -i
                 PAGES        TOTAL      PERCENTAGE
    TOTAL MEM  261010406     995.7 GB         ----
         FREE  228447706     871.5 GB   87% of TOTAL MEM
         USED  32562700     124.2 GB   12% of TOTAL MEM
       SHARED  1626752       6.2 GB    0% of TOTAL MEM
      BUFFERS     3277      12.8 MB    0% of TOTAL MEM
       CACHED  20211378      77.1 GB    7% of TOTAL MEM
         SLAB   765701       2.9 GB    0% of TOTAL MEM

   TOTAL HUGE        0            0         ----
    HUGE FREE        0            0    0% of TOTAL HUGE

   TOTAL SWAP  1048575         4 GB         ----
    SWAP USED        0            0    0% of TOTAL SWAP
    SWAP FREE  1048575         4 GB  100% of TOTAL SWAP

 COMMIT LIMIT  131553778     501.8 GB         ----
    COMMITTED  26620006     101.5 GB   20% of TOTAL LIMIT
```

* Running the `sys -i` command shows more details about the hardware, as well as BIOS information, and product tags. Sensitive information such as serial numbers have been removed in the following example:

```bash
crash> sys -i
        DMI_BIOS_VENDOR: HPE
       DMI_BIOS_VERSION: A43
          DMI_BIOS_DATE: 12/03/2021
         DMI_SYS_VENDOR: HPE
       DMI_PRODUCT_NAME: ProLiant DL325 Gen10 Plus
    DMI_PRODUCT_VERSION:
     DMI_PRODUCT_SERIAL: -
       DMI_PRODUCT_UUID: -
       DMI_BOARD_VENDOR: HPE
         DMI_BOARD_NAME: ProLiant DL325 Gen10 Plus
      DMI_BOARD_VERSION:
       DMI_BOARD_SERIAL: -
    DMI_BOARD_ASSET_TAG: -
     DMI_CHASSIS_VENDOR: HPE
       DMI_CHASSIS_TYPE: 23
    DMI_CHASSIS_VERSION:
     DMI_CHASSIS_SERIAL: -
  DMI_CHASSIS_ASSET_TAG: -
```

* The `bt` command produces a backtrace output for the process that caused the kernel panic. The succeeding example shows process `650` running command `khungtaskd` was responsible for the kernel panic:

```bash
crash> bt
PID: 650      TASK: ffff9b2ffbb59080  CPU: 47   COMMAND: "khungtaskd"
 #0 [ffff9b2ffb663cb0] machine_kexec at ffffffffb2c69854
 #1 [ffff9b2ffb663d10] __crash_kexec at ffffffffb2d29f12
 #2 [ffff9b2ffb663de0] panic at ffffffffb33ab713
 #3 [ffff9b2ffb663e60] watchdog at ffffffffb2d56bfe
 #4 [ffff9b2ffb663ec8] kthread at ffffffffb2ccb621
```

* More information on `khungtaskd` can be found [at this article](https://www.tencentcloud.com/document/product/213/41974), however a summary can be observed beneath:

> The kernel hung task is based on a single kernel thread named as khungtaskd, which monitors processes in the TASK_UNINTERRUPTIBLE status. If a process stuck in D state during the period specified by kernel.hung_task_timeout_secs (defaults to 120 seconds), the stack information of this hung task process will be printed.

* To verify that the kernel is configured to crash in a `khungtaskd` state, the `p` command (to print the value of an expression) with the `sysctl_hung_task_panic` argument can be used:

```bash
crash> p sysctl_hung_task_panic
sysctl_hung_task_panic = $1 = 1
```

* Similarly the timeout can also be checked using the `p sysctl_hung_task_timeout_secs` command:

```bash
crash> p sysctl_hung_task_timeout_secs
sysctl_hung_task_timeout_secs = $2 = 360
```

* To view more information about a particular process, the `ps` command plus the PID number as an argument can be used:

```bash
crash> ps 650
      PID    PPID  CPU       TASK        ST  %MEM      VSZ      RSS  COMM
>     650       2  47  ffff9b2ffbb59080  RU   0.0        0        0  [khungtaskd]
```

* Finding out more information about which tasks were blocked, can be done with the `log` command and a `grep` for the word `INFO`. In the subsequent example, `systemd` is seen as being blocked:

```bash
crash> log | grep INFO
[3970064.996286] INFO: task systemd:1 blocked for more than 360 seconds.
```

* For showing the stack traces of all blocked tasks in a `TASK_UNINTERRUPTIBLE` state, the `foreach UN bt` command is recommended:

```bash
foreach UN bt
```

* Further filtering is also possible on all blocked tasks with this command string. The command picks tasks #2, #3, and #4 from each process, places them into three columns, counts the amount of stack trace messages occurrences and sorts from highest to lowest:

```bash
foreach UN bt | awk '/#[2-4] /{print $3,$5}' | paste - - - | sort | uniq -c | sort -nr
```

* An example is here:

```bash
crash> foreach UN bt | awk '/#[2-4] /{print $3,$5}' | paste - - - | sort | uniq -c | sort -nr
     46 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     proc_cgroup_show ffffffffb2d35146
     33 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     cgroup_lock_live_group ffffffffb2d2ed49
      2 __mutex_lock_killable_slowpath ffffffffb33b6520 mutex_lock_killable ffffffffb33b65fe    iterate_dir ffffffffb2e71d00
      1 schedule_timeout ffffffffb33b5831       wait_for_completion ffffffffb33b805d    wait_rcu_gp ffffffffb2cc828e
      1 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     proc_cgroupstats_show ffffffffb2d2f2be
      1 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     cgroup_release_agent ffffffffb2d2ffe5
      1 __mutex_lock_slowpath ffffffffb33b6ca7  mutex_lock ffffffffb33b602f     cgroup_free_fn ffffffffb2d34308
```

* From the above output, a high volume of `mutex_lock_slowpath` messages can be viewed. Researching the error attributes this to either a hardware issue or low resource availability (CPU / Memory / I/O).

* The full list of unique blocked tasks can be found by running `foreach UN bt | grep COMMAND | awk '{print $NF}' | sort -u`:

```bash
crash> foreach UN bt | grep COMMAND | awk '{print $NF}' | sort -u
"cadvisor"
"dockerd"
"filebeat"
"kubelet"
"kworker/113:0"
"kworker/31:1"
"process-exporte"
"runc"
"systemd"
"systemd-journal"
```

* To dig further into a blocked task, use the `set` command with the PID as the argument. The ensuing example sets the focus on the blocked task `systemd`:

```bash
crash> set 1
    PID: 1
COMMAND: "systemd"
   TASK: ffff9a32db050000  [THREAD_INFO: ffff9a32dbb9c000]
    CPU: 42
  STATE: TASK_UNINTERRUPTIBLE
```

* Then run `bt -l` for more verbose backtrace messages. In the above-mentioned `mutex_lock_slowpath` example, the process that was running before the `mutex_lock` was engaged was `proc_cgroup_show`:

```bash
crash> bt -l
#2 [ffff9a32dbb9fda8] __mutex_lock_slowpath at ffffffffb33b6ca7
    /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/include/linux/spinlock.h: 337
#3 [ffff9a32dbb9fe08] mutex_lock at ffffffffb33b602f
    /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/arch/x86/include/asm/current.h: 14
#4 [ffff9a32dbb9fe20] proc_cgroup_show at ffffffffb2d35146
    /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/kernel/cgroup.c: 4706
```

* The line number from the `cgroup.c` file being referenced by `proc_cgroup_show` was `4706`. It is possible to view the relevant source code from the installed `kernel-debuginfo` packages with the `list` or `l` command:

```bash
crash> l /usr/src/debug/kernel-3.10.0-1160.119.1.el7/linux-3.10.0-1160.119.1.el7.x86_64/kernel/cgroup.c: 4706
4701
4702            retval = 0;
4703
4704            mutex_lock(&cgroup_mutex);
4705
4706            for_each_active_root(root) {
4707                    struct cgroup_subsys *ss;
4708                    struct cgroup *cgrp;
4709                    int count = 0;
4710
```

* You can see from line `4704` that the `mutex_lock` is related to the `Cgroup` system. From the [kernel.org documentation on Cgroups](https://docs.kernel.org/admin-guide/cgroup-v1/cgroups.html):

> There is a global mutex, cgroup_mutex, used by the cgroup system. This should be taken by anything that wants to modify a cgroup. It may also be taken to prevent cgroups from being modified, but more specific locks may be more appropriate in that situation.

* Since `Cgroups` handle the allocation of resources, the mutex lock observed here points towards a lack of resources on the node.

* As an aside, if the `mutex_lock` code pointed to `mutex_lock(&dentry->d_inode->i_mutex);`, that would highlight an inode-related error on the storage medium and further investigation would be required there.

* Finally, after running the `log` command to dump the `system message buffer`, multiple `Hardware Error` messages were observed related to the customer's memory DIMMs:

```bash
crash> log
<output_redacted>
[3968484.124456] {176586}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 20480
[3968484.124460] {176586}[Hardware Error]: It has been corrected by h/w and requires no further action
[3968484.124461] {176586}[Hardware Error]: event severity: corrected
[3968484.124463] {176586}[Hardware Error]:  Error 0, type: corrected
[3968484.124464] {176586}[Hardware Error]:  fru_text: DIMM# Sourced
[3968484.124465] {176586}[Hardware Error]:   section_type: memory error
[3968484.124466] {176586}[Hardware Error]:   error_status: 0x0000000000040400
[3968484.124468] {176586}[Hardware Error]:   physical_address: 0x000000f877d52000
[3968484.124470] {176586}[Hardware Error]:   node: 0 card: 3 module: 1 rank: 2 bank: 6 row: 57311 column: 0
[3968484.124471] {176586}[Hardware Error]:  Error 1, type: corrected
[3968484.124472] {176586}[Hardware Error]:  fru_text: DIMM# Sourced
[3968484.124473] {176586}[Hardware Error]:   section_type: memory error
[3968484.124474] {176586}[Hardware Error]:   error_status: 0x0000000000040400
[3968484.124475] {176586}[Hardware Error]:   physical_address: 0x000000f7f9b63180
[3968484.124477] {176586}[Hardware Error]:   node: 0 card: 3 module: 1 rank: 2 bank: 6 row: 56806 column: 16
[3968484.124501] [Hardware Error]: Corrected error, no action required.
[3968484.125383] [Hardware Error]: CPU:0 (17:31:0) MC1_STATUS[-|CE|-|-|AddrV|-|-|-|-]: 0x940000000000009f
[3968484.126221] [Hardware Error]: Error Addr: 0x000000f877d52000, IPID: 0x0000000000000000
[3968484.126907] [Hardware Error]: Instruction Fetch Unit Extended Error Code: 0
[3968484.127593] [Hardware Error]: Instruction Fetch Unit Error: microtag probe port parity error.
[3968484.128280] [Hardware Error]: cache level: L3/GEN, tx: RESV
```

## Example issue resolution

Ultimately it was found that the lack of resources were due to `Hardware Errors` in the customer's memory DIMMs, thus causing a contention for resources resulting in tasks being blocked for over 360 seconds and therefore invoking a kernel panic.

## Conclusion

`vmcore` dump analysis is a vast and complicated process. There are almost a limitless number of ways that the kernel can crash and the root cause may not always be the same from system to system. The author hopes that the example analysis above for blocked tasks can help to demystify some of the research required and make it easier to identify a root cause if you run into a similar situation.


Tips and Tricks for Analyzing a vmcore dump with the Crash Utility


Selecting [Azure Confidential Computing](https://learn.microsoft.com/en-us/azure/confidential-computing/confidential-vm-faq) (ACC) for your workload gives you exceptional security for sensitive workloads on Rocky Linux from CIQ. However, with that added protection comes a few critical things to keep in mind, especially if you're using `Confidential OS Disk Encryption`.

This guide will help you avoid common issues that can lead to inaccessible VMs and downtime. Prevention is simple once you know what to look out for.

## Why this matters

When using encrypted disks, the encryption key is sealed to specific system values (PCRs) during provisioning. These values depend on things like:

* The UEFI firmware

* The Secure Boot shim

* Kernel certificates and signatures

Any unexpected change to these components, even valid ones, can break the key seal. If that happens, your system will fail to unlock the disk automatically, and you'll be stuck at a boot time passphrase prompt via the serial console.

## What not to change

To keep your system booting normally, avoid making changes to Secure Boot related components unless absolutely necessary:

### Don't change the shim

[Rocky Linux from CIQ](https://ciq.com/products/rocky-linux/) uses a `shim` signed by Microsoft and provided by CIQ. Replacing it or using a community version will break the chain of trust.

### Don’t install a custom kernel

Only use kernels installed by default with your CIQ image. Custom kernels, even from Rocky Linux's upstream (RESF) or [kernel.org](https://www.kernel.org/), will likely break the boot process.

### Avoid untrusted drivers or modules

Some drivers or changes to the initrd can affect the PCRs, even if the system still appears secure.

### Set a fallback passphrase

To avoid being permanently locked out, add a custom passphrase to the TPM:

```bash
cryptsetup luksAddKey /dev/sdX --token-type systemd-tpm2
```

Replace `/dev/sdX` with your encrypted disk (use `lsblk` to find it). Store the passphrase somewhere safe.

You could test it by forcing a PCR change:

```bash
cd /boot/efi/EFI/rocky
mv shim_certificate.efi.bak shim_certificate.efi
reboot
```

Then use the passphrase in the serial console when prompted. After testing, restore the original `shim`:

```bash
mv shim_certificate.efi shim_certificate.efi.bak
```

## Notes

Consider creating a separate recovery key. This would allow you to unlock the encrypted volume by attaching the VHD to another VM, handy for emergencies.

Always create your fallback passphrase as one of your first steps when deploying a new instance.

Avoid unsupported kernels or unauthorized Secure Boot changes.

Stick with 5th or 6th gen DC- and EC- Azure instance types for full ACC support.

Being proactive helps you stay secure without getting locked out.

## References & related articles

[Azure Confidential Computing](https://learn.microsoft.com/en-us/azure/confidential-computing/confidential-vm-faq)  
[Data & System Recovery Considerations on Azure Confidential Computing (ACC)](https://ciq.com/blog/data-and-system-recovery-considerations-on-azure-confidential-computing-acc/)


Preventing Boot Issues on Azure Confidential Computing with Rocky Linux


## Introduction

This article addresses an issue where a Rocky Linux server fails to boot and is waiting on mount points. The system drops to emergency mode because LVM physical volumes are not detected, even though some logical volumes remain available.

## Problem

After applying system patches, the server fails to boot normally. The OS drops to emergency mode, reporting missing logical volumes (such as `/var` and `/tmp`). LVM commands do not detect any physical volumes, though some logical volumes (like root and swap) may still be available.

## Symptoms

- System fails to boot into multi-user mode and drops to emergency mode.
- Boot process hangs waiting for logical volumes.
- LVM commands (`pvs`, `pvscan`, `lvmdiskscan`) return no results.
- Some logical volumes are available and mountable.
- When booted from rescue media, `lsblk` shows the logical volumes, but LVM commands still show nothing.

## Resolution

### Boot into Rescue Mode

Boot from any Rocky Linux installation ISO and select `Troubleshooting` from the boot menu. 

Choose `Rescue a Rocky Linux system`. 

Once the new environment is loaded, it will ask if you would like to automatically detect an existing installation. Press `1` to continue to boot into a shell.

### Verify LVM State

Run the following commands to check the state of LVM:

```shell
lvs -a -o +devices
pvdisplay
lsblk
```

If `lsblk` shows the logical volumes but LVM commands do not, proceed to the next step.

### Rebuild LVM Device Settings

Back up the current LVM devices file:

```shell
mv /etc/lvm/devices/system.devices /root/system.devices.backup
```

Recreate LVM device settings:

```shell
vgimportdevices -a
```

Reboot the system:

```shell
reboot
```

After rebooting, the system should detect all LVM physical volumes and boot normally.

## Root Cause

This issue is typically caused by a corrupted or outdated `/etc/lvm/devices/system.devices` file, which prevents LVM from detecting physical volumes after system changes such as patching or kernel updates. Rebuilding the device settings allows LVM to rescan and recognize the correct devices.

## Notes

- LVM backup metadata is stored in `/etc/lvm/backup/` and `/etc/lvm/archive/`. Always ensure these backups are preserved before making changes.
- Removing or moving the `system.devices` file is safe as long as a backup is created.

## References

[Rocky Linux LVM Documentation](https://docs.rockylinux.org/ja/books/admin_guide/07-file-systems/)  
[vgimport man page](https://man7.org/linux/man-pages/man8/vgimportdevices.8.html)


LVM Physical Volumes Not Detected in Rocky Linux


## Introduction

This article outlines the steps required to configure a PXE boot environment for Rocky Linux using Lorax to generate boot image and the Rocky Linux DVD image for creating the BaseOS and AppStream repositories.

This guide uses `httpd` for serving the installation tree, `dnsmasq` to provide DHCP and TFTP, and `lorax` to generate EFI bootable installation media.

This setup enables automated network-based Rocky Linux installations over a local network.

## Problem

Users often need to deploy Rocky Linux at scale without manually installing each system.

Manual DVD-based installation is not feasible in large environments or headless setups.

There is a need to set up a PXE boot environment that supports UEFI systems and provides the Rocky Linux installer via network boot.

## Symptoms

Client machines are unable to PXE boot due to missing or misconfigured TFTP, HTTP, or DHCP services.

A boot fails with errors such as "no DHCP response", "file not found via TFTP", or "missing installation source".

UEFI systems may hang or drop to a shell if the GRUB configuration or bootloader path is incorrect.

## Resolution

### Prerequisites

* `root` access or `sudo` privileges.

* A Rocky Linux DVD ISO.

* Two machines, one as the PXE host and one as the PXE client directly connected to each other / on a segmented network and doesn't have DHCP servers running.

* A working Internet connection.

### Initial setup

* Install these required packages:

```bash
sudo dnf install -y lorax httpd tftp-server dnsmasq
```

* Enable the `httpd` and `tftp` services:

```bash
sudo systemctl enable --now httpd
sudo systemctl enable --now tftp
```

* Download the Rocky Linux 10 DVD ISO (Rocky Linux 10 is the example used throughout this article):

```bash
wget https://download.rockylinux.org/pub/rocky/10/isos/x86_64/Rocky-10.1-x86_64-dvd1.iso
```

* Mount the ISO and copy its contents:

```bash
mkdir /mnt/dvd
sudo mount -o loop ~/Rocky-10.1-x86_64-dvd1.iso /mnt/dvd
sudo mkdir -p /var/www/html/rocky10-dvd
sudo cp -av /mnt/dvd/. /var/www/html/rocky10-dvd/
```

### Boot image generation

* Generate the Rocky Linux 10 boot image:

```bash
sudo lorax --product "RockyLinux" --version "10" --release "10" --source "https://download.rockylinux.org/pub/rocky/10/BaseOS/x86_64/os/" --source "https://download.rockylinux.org/pub/rocky/10/AppStream/x86_64/os/" --isfinal --logfile lorax.log --buildarch x86_64 --volid "RL10_LIVENET" /tmp/lorax-out
```

* Copy the PXE boot files:

```bash
sudo mkdir -p /var/lib/tftpboot/rocky10
sudo cp /tmp/lorax-out/images/pxeboot/{vmlinuz,initrd.img} /var/lib/tftpboot/rocky10/
```

* Copy the installer stage 2 files:

```bash
sudo cp -r /tmp/lorax-out/images/ /var/www/html/rocky10/
```

* Mount the EFI boot image:

```bash
mkdir /tmp/efiboot
sudo mount -o loop /tmp/lorax-out/images/efiboot.img /tmp/efiboot
```

* Copy UEFI boot files:

```bash
sudo mkdir -p /var/lib/tftpboot/rocky10/EFI/BOOT
sudo cp -av /tmp/efiboot/EFI/BOOT/* /var/lib/tftpboot/rocky10/EFI/BOOT/
```

### Grub configuration

* Configure GRUB:

```bash
cat << "EOF" | sudo tee /var/lib/tftpboot/rocky10/EFI/BOOT/grub.cfg
menuentry 'Rocky Linux 10 Install (PXE Boot)' {
    linuxefi /rocky10/vmlinuz inst.stage2=http://192.168.1.150/rocky10 inst.repo=http://192.168.1.150/rocky10-dvd
    initrdefi /rocky10/initrd.img
}
EOF
```

* Configure `dnsmasq` for PXE boot:

```bash
cat << "EOF" | sudo tee /etc/dnsmasq.conf
interface=enp2s0
bind-interfaces
domain-needed
bogus-priv
log-dhcp
dhcp-range=192.168.1.2,192.168.1.249,12h
dhcp-boot=rocky10/EFI/BOOT/grubx64.efi
enable-tftp
tftp-root=/var/lib/tftpboot
EOF
```

* Restart services:

```bash
sudo systemctl restart dnsmasq
sudo systemctl restart httpd
```

* Allow services through the firewall:

```bash
sudo firewall-cmd --add-service=tftp --permanent
sudo firewall-cmd --add-service=http --permanent
sudo firewall-cmd --add-service=dns --permanent
sudo firewall-cmd --add-service=dhcp --permanent
sudo firewall-cmd --reload
```

* Disable SELinux:

```bash
sudo setenforce 0
sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/' /etc/selinux/config
```

## Root cause

PXE boot environments require coordination between DHCP, TFTP, and HTTP services.

Incorrect or missing files, misconfigured GRUB boot paths, or firewall/SELinux restrictions can prevent PXE booting from succeeding.

Without properly generated and served installer images and boot files, UEFI systems cannot retrieve necessary components to launch the installer.

## Notes

This guide is written for UEFI PXE clients only.

Legacy BIOS support would require additional configuration changes to use `pxelinux.0` or GRUB BIOS equivalents.

Ensure that the `interface=` line in `dnsmasq.conf` matches your actual network interface name.

Network clients must be on the same subnet as the PXE server.

## References & related articles

[Creating a Custom Rocky Linux ISO](https://docs.rockylinux.org/guides/isos/iso_creation/)


PXE Boot Installation of Rocky Linux Using Lorax, dnsmasq, and TFTP


## Introduction

CentOS 7 became [End of Life on June 30th, 2024](https://blog.centos.org/2023/04/end-dates-are-coming-for-centos-stream-8-and-centos-linux-7/) and is now no longer supported with any security updates of any kind. In order to make sure that your systems are up-to-date, switching to Rocky Linux is an excellent choice for security and peace of mind.

If you need more time before starting the migration over to Rocky Linux, then [CIQ Bridge](https://ciq.com/products/ciq-bridge/) is the recommended solution for you.

## Prerequisites

- A CentOS 7.9 node to perform the migration on.

---

**_⚠️ WARNING_** _Make sure you are on the latest minor version of CentOS 7 (`7.9`), before attempting the migration._

---

---

**_⚠️ WARNING_** _You are not able to directly go from CentOS 7.9 to Rocky Linux 9.x. You must upgrade to Rocky Linux 8.10. Upgrades between major versions of Rocky Linux (8.x to 9.x) using Leapp [are not recommended](https://docs.rockylinux.org/release_notes/9_7/) and it is better to perform a fresh install._

---

---

**_⚠️ WARNING_** _Ensure that all data is safely backed up in at least three locations, before the migration begins._

---

---

**_⚠️ WARNING_** _While Leapp is the tool being championed in this article, the best method for migration that CIQ recommends is to set up a new node with Rocky Linux 9.7, move your data and applications over to that node, and then perform a cut off when the CentOS 7.9 node is no longer required. Even in a basic CentOS 7.9 migration without any specific applications installed, many blockers can occur during the migration process._

---

- Run the below commands as either `root` or a user with `sudo` privileges:

## Updating the CentOS 7.9 repositories to point towards CentOS Vault

- Overwrite the `CentOS-Base` repo file in `/etc/yum.repos.d/CentOS-Base.repo` with the following:

```bash
cat << "EOF" | sudo tee /etc/yum.repos.d/CentOS-Base.repo
[base]
name=CentOS-$releasever - Base
baseurl=http://vault.centos.org/7.9.2009/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[updates]
name=CentOS-$releasever - Updates
baseurl=http://vault.centos.org/7.9.2009/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[extras]
name=CentOS-$releasever - Extras
baseurl=http://vault.centos.org/7.9.2009/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[centosplus]
name=CentOS-$releasever - Plus
baseurl=http://vault.centos.org/7.9.2009/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
EOF
```

- Upgrade all packages:

```bash
yum upgrade -y
```

- Reboot:

```bash
reboot
```

## Leapp installation

- Set up the ELevate repository:

```bash
yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el$(rpm --eval %rhel).noarch.rpm
```

- Install the `leapp` packages:

```bash
yum install -y leapp-upgrade leapp-data-rocky
```

- Run pre-upgrade checks:

```bash
leapp preupgrade
```

- In the event the `preupgrade` process highlights any issues, check the `/var/log/leapp/leapp-report.txt` file for a list of all issues and `/var/log/leapp/leapp-preupgrade.log` for a full log from the `preupgrade` process.

- In `leapp-report.txt`, what you want to look for is `Upgrade has been inhibited due to the following problems`. The items listed underneath that will prevent the upgrade from starting, even if you run the `leapp upgrade` command.

The following section presents examples of blockers and how they can be addressed:

### Example blockers

#### Missing required answers in the answer file

```bash
Risk Factor: high (inhibitor)
Title: Missing required answers in the answer file
Summary: One or more sections in answerfile are missing user choices: remove_pam_pkcs11_module_check.confirm
For more information consult https://red.ht/leapp-dialogs.
Related links:
    - Leapp upgrade fail with error "Inhibitor: Missing required answers in the answer file.": https://access.redhat.com/solutions/7035321
Remediation: [hint] Please register user choices with leapp answer cli command or by manually editing the answerfile.
[command] leapp answer --section remove_pam_pkcs11_module_check.confirm=True
```

##### Solution

- The `remove_pam_pkcs11_module_check` check in the `/var/log/leapp/answerfile` file has to be set to `True`:

```bash
leapp answer --section remove_pam_pkcs11_module_check.confirm=True
```

- Once done, run the `leapp preupgrade` command again and if there are no other `inhibitors` found, the end summary will become either yellow or green, indicating that the upgrade can proceed.

#### Detected custom leapp actors or files

```bash
Risk Factor: high
Title: Detected custom leapp actors or files.
Summary: We have detected installed custom actors or files on the system. These can be provided e.g. by third party vendors, Red Hat consultants, or can be created by users to customize the upgrade (e.g. to migrate custom applications). This is allowed and appreciated. However Red Hat is not responsible for any issues caused by these custom leapp actors. Note that upgrade tooling is under agile development which could require more frequent update of custom actors.
The list of custom leapp actors and files:
    - /usr/share/leapp-repository/repositories/system_upgrade/common/files/rpm-gpg/8/RPM-GPG-KEY-Rocky-8
Related links:
    - Customizing your Red Hat Enterprise Linux in-place upgrade: https://red.ht/customize-rhel-upgrade
Remediation: [hint] In case of any issues connected to custom or third party actors, contact vendor of such actors. Also we suggest to ensure the installed custom leapp actors are up to date, compatible with the installed packages.
```

##### Solution

- This can be safely ignored.

#### GRUB2 core will be automatically updated during the upgrade

```bash
Risk Factor: high
Title: GRUB2 core will be automatically updated during the upgrade
Summary: On legacy (BIOS) systems, GRUB2 core (located in the gap between the MBR and the first partition) cannot be updated during the rpm transaction and Leapp has to initiate the update running "grub2-install" after the transaction. No action is needed before the upgrade. After the upgrade, it is recommended to check the GRUB configuration.
```

##### Solution

- This can also safely be ignored if you are on a UEFI system.

#### Difference in Python versions and support in RHEL 8

```bash
Risk Factor: high
Title: Difference in Python versions and support in RHEL 8
Summary: In RHEL 8, there is no 'python' command. Python 3 (backward incompatible) is the primary Python version and Python 2 is available with limited support and limited set
of packages. If you no longer require Python 2 packages following the upgrade, please remove them. Read more here: https://red.ht/rhel-8-python
Related links:
    - Difference in Python versions and support in RHEL 8: https://red.ht/rhel-8-python
Remediation: [hint] Please run "alternatives --set python /usr/bin/python3" after upgrade
```

##### Solution

- Move your applications over to using Python 3.

- The Python 2 packages have many other packages that depend on them. The recommendation is to not remove these packages, unless absolutely required.

#### Risk factor: low warnings

- Go through all of these, however usually these can be ignored (one of the warnings, is that `SELinux` will be set to `permissive` mode, so make sure to change that back to `enforcing` if needed):

```bash
sed -i 's/SELINUX=permissive/SELINUX=enforcing/' /etc/selinux/config

setenforce 1
```

## Start the upgrade

- Once the above `inhibitor` issues have been sorted, run this command to start the upgrade process:

```bash
leapp upgrade
```

- Once completed, `reboot` the machine:

```bash
reboot
```

- At the `GRUB` menu, select the `ELevate-Upgrade-Initramfs` option.

- When that section of the installation is complete, the system will reboot into Rocky Linux (you will see the `grub` menu populated with the Rocky Linux kernels).

- `SELinux` will then perform a relabel upon first boot.

- Once the relabel is complete, the system will reboot a second time.

- If all is successful, you will be presented with a login prompt and at the top you will see `Rocky Linux 8.10 (Green Obsidian)`.

- Check for any leftover packages from the migration:

```bash
rpm -qa | grep "el7\."
```

- Remove these with the `dnf remove` command.

## References & related articles

[Rocky Linux Forum User's Experience Upgrading from Rocky Linux 8 to Rocky Linux 9](https://forums.rockylinux.org/t/update-rocky-linux-8-to-9/10338)
[Rocky Linux 9.7 Release Documentation](https://docs.rockylinux.org/release_notes/9_7/)


How to Migrate From CentOS 7.9 to Rocky Linux 8.10 with Leapp

How to Migrate From CentOS 8.5 to Rocky Linux 9.5 with Leapp


## Introduction

Customers often need a custom ISO for a litany of reasons - perhaps they require a mirror of a repository or to edit kernel cmdline parameters for their own customers.

## Problem

This guide explains how to create a custom Rocky Linux 8.10 installation ISO.

## Symptoms

The customer did not have clear direction or resources on how to build their own ISO.

## Resolution

- This guide includes a high-level overview of the ISO build process, along with the commands needed to generate a custom ISO.
- A guide on how to create a custom Rocky Linux ISO was also pushed upstream to the [Rocky Linux Docs](https://docs.rockylinux.org/10/guides/isos/iso_creation/). Please find the same guide below:

### Prerequisites

- A 64 bit machine running Rocky Linux 9 to build the new ISO image.
- A Rocky Linux 9 DVD ISO image.
- A `kickstart` file to apply to the ISO.
- Read the Lorax [Quickstart](https://weldr.io/lorax/lorax.html#quickstart) and [mkksiso](https://weldr.io/lorax/mkksiso.html) documentation to become familiar with how the `Anaconda` `boot.iso` is created.

### Package installation and setup

- Install the `lorax` package:

```bash
sudo dnf install -y lorax
```

### Building the ISO with a kickstart file

- Run the `mkksiso` command to add a `kickstart` file and then build a new ISO:

```bash
mkksiso --ks <PATH_TO_KICKSTART_FILE> <PATH_TO_ISO_TO_MODIFY> <OUTPUT_PATH_FOR_BUILT_ISO>
```

- Below is an example `kickstart` file `example-ks.cfg`, which sets up a Rocky Linux 9.5 `Server With GUI` environment:

```text
lang en_GB
keyboard --xlayouts='us'
timezone Asia/Tokyo --utc
reboot
cdrom
bootloader --append="rhgb quiet crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
zerombr
clearpart --all --initlabel
autopart
network --bootproto=dhcp
firstboot --disable
selinux --enforcing
firewall --enabled
%packages
@^server-product-environment
%end
```

### Adding a repository with its packages to an ISO image

- Make sure the repository you want to add has the `repodata` directory inside of it. If not, this can be created using the `createrepo_c` command and this can be installed with `sudo dnf install -y createrepo_c`
- Add the repository to your `kickstart` file, using the following syntax:

```text
repo --name=extra-repo --baseurl=file:///run/install/repo/<YOUR_REPOSITORY>/
```

- Add your repository using the `--add` flag with the `mkksiso` tool:

```bash
mkksiso --add <LINK_TO_YOUR_REPOSITORY> --ks <PATH_TO_KICKSTART_FILE> <PATH_TO_ISO_TO_MODIFY> <OUTPUT_PATH_FOR_BUILT_ISO>
```

- The process is detailed further using the `baseos` repository in the example below:
- The `baseos` repository will be locally downloaded along with all of its packages:

```bash
dnf reposync -p ~ --download-metadata --repo=baseos
```

- Then add the repository to the `kickstart` file:

```text
repo --name=extra-repo --baseurl=file:///run/install/repo/baseos/
```

- The `kickstart` file would look like the following:

```text
lang en_GB
keyboard --xlayouts='us'
timezone Asia/Tokyo --utc
reboot
cdrom
bootloader --append="rhgb quiet crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
zerombr
clearpart --all --initlabel
autopart
network --bootproto=dhcp
firstboot --disable
selinux --enforcing
firewall --enabled
repo --name=extra-repo --baseurl=file:///run/install/repo/baseos/
%packages
@^server-product-environment
%end
```

- Then point the `mkksiso` command directly to the repository directory and build the ISO:

```bash
mkksiso --add ~/baseos --ks example-ks.cfg ~/Rocky-9.5-x86_64-dvd.iso ~/Rocky-9.5-x86_64-dvd-new.iso
```

### Conclusion

What has been discussed here are just a few of the options available to tweak and build your own Rocky Linux ISO. For further ways, including modifying the kernel cmdline arguments, the author highly recommends to go through the [mkksiso](https://weldr.io/lorax/mkksiso.html) documentation in more detail.

## References & related articles

[Anaconda Documentation](https://anaconda-installer.readthedocs.io/en/latest/)\
[Automating the Installation with Kickstart - Fedora Wiki](https://docs.fedoraproject.org/en-US/fedora/f36/install-guide/advanced/Kickstart_Installations/#chap-kickstart-installations)\
[Dracut - Fedora Wiki](https://fedoraproject.org/wiki/Dracut)\
[Lorax Documentation](https://weldr.io/lorax/)\
[mkksiso Documentation](https://weldr.io/lorax/mkksiso.html)\
[Mock Repository](https://github.com/rpm-software-management/mock)\
[Peridot Repository](https://github.com/rocky-linux/peridot)\
[Creating a Custom Rocky Linux ISO - Rocky Linux Docs](https://docs.rockylinux.org/10/guides/isos/iso_creation/)


Creating a Custom Rocky Linux ISO


## Introduction

This guide explains how to synchronize repositories locally using `reposync`. It covers syncing both publicly available repositories and private repositories, such as those within CIQ Depot. This guide will provide you with two options for downloading packages depending on the complexity of your needs.

## Problem

You need to create a local copy of a repository for an air-gapped environment, a network with limited or slow internet access, or to provide faster access to a local set of servers.

## Prerequisites

A web server is required to serve the downloaded packages. You can follow our [guide](https://ciq.com/blog/how-to-set-up-a-web-server-in-rocky-linux/) on how to set up a web server. When syncing repositories, you will also need to consider the storage requirements.

For smaller repositories that only feature updates such as our [Long Term Support (LTS)](https://ciq.com/services/long-term-support/) repository, the overall size can be somewhere in the range of 1GB to 15GB or more in size.

The full repository for Rocky Linux or CentOS can potentially exceed 100GB in total size.

In this guide, the web server will have an IP address of `192.168.1.30`.

## Option 1: Sync on an existing server

If you are using a Rocky Linux server or another server using CIQ's [CentOS Bridge service](https://ciq.com/products/ciq-bridge/) that is already configured for the repository you are looking to sync, you can leverage its existing setup to download the repository.

Identify the repository you want to download by running `dnf repolist`. The first column will provide you with the repo id that you will use in your reposync command.

In this example, CIQ is syncing using the `repoid` of *rocky-lts-9.2.x86_64* attached to our *CIQ LTS for Rocky Linux 9.2* repository.

```bash
dnf reposync -p repodata --download-metadata \
  --repoid=rocky-lts-9.2.x86_64 --download-path=/var/www/html/rocky-lts-9.2.x86_64/
```

If you are trying to sync our [CentOS Bridge](https://depot.ciq.com/my/bridge/) repository (requires a subscription to CIQ Depot) from a CentOS 7 server, the command is different:

```bash
reposync -p repodata --download-metadata \
   --repoid=ciq-bridge.x86_64 --download_path=/var/www/html/ciq-bridge/
```

The above commands will store the repository files in the local web server path `/var/www/html/`. You can also store these files on shared storage for centralized access or copy them to another server.

## Option 2: Sync on a separate server

If you want to run this on a dedicated server or need to sync multiple repositories, consider creating a dedicated repo file(s) to manage the information.

### Step 1: Create a repository configuration file

If you plan to sync CIQ Depot repositories, you can obtain the necessary repo information from the following location:

1. Navigate to [My Products](https://depot.ciq.com/my/).
2. Select the repository you would like to sync.
3. Click the button with the three dots on the right-hand side of the repository name.

4. Select **"DNF Repo Config"**.

For our example, we will use the publicly available Rocky Linux 9 BaseOS repository. We will create a new repository file named `rocky-baseos-9.repo` in our root directory. Feel free to save this in a more permanent location.

```ini
[rocky-baseos-9]
name=Rocky Linux 9 - BaseOS
baseurl=http://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/os/
gpgcheck=1
enabled=1
countme=1
metadata_expire=6h
gpgkey=https://dl.rockylinux.org/pub/rocky/RPM-GPG-KEY-Rocky-9
```

The repo id is specified at the top of the file between the `[]` characters. In this case, `[rocky-baseos-9]`.

Repeat the above step to create separate files for each repository you plan to sync.

Alternatively, you can add multiple repositories to a single file, ensuring each has a unique repo id.

### Step 2: Run `reposync`

Once the configuration file is created, you can run the `reposync` command as before. Only now you will specify the repo file like in the below example:

```bash
dnf -c ./rocky-baseos-9.repo reposync -v -p repodata \
--download-metadata --repoid=rocky-baseos-9 \
--download-path=/var/www/html/rocky-baseos-9/
```

Repeat this step for each file and/or repository you intend to sync ensuring that you change the file, repo id, and folder as needed.

### Step 3: Verify and configure clients

After all downloads are complete, confirm the repository is accessible via your web browser.

In the below example, our repository is located at `http://192.168.1.30/rocky-baseos-9/rocky-baseos-9/`.

Once you’ve verified that the packages are being served correctly, configure Rocky Linux to use the repository by updating the appropriate file in `/etc/yum.repos.d/`.

As illustrated below, since we are serving the BaseOS repository, we will modify `/etc/yum.repos.d/rocky.repo`.

```ini
[baseos]
name=Rocky Linux $releasever - BaseOS
#mirrorlist=https://mirrors.rockylinux.org/mirrorlist?arch=$basearch&repo=BaseOS-$releasever$rltype
baseurl=http://192.168.1.30/rocky-baseos-9/rocky-baseos-9/
gpgcheck=1
enabled=1
countme=1
metadata_expire=6h
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Rocky-9
```

## Notes

### Automate reposync

There are several ways to automate fetching the latest packages via `reposync`.

For a complete, centralized solution, we recommend running a playbook within [Ascender](https://ciq.com/products/ascender/).

Alternatively, you can set up a simple [cronjob](https://docs.rockylinux.org/guides/automation/cron_jobs_howto/) to pull the latest packages automatically. For example, you might create `/etc/cron.d/reposync` in the example provided:

```bash
# Run the hourly jobs
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
@daily root dnf -c /root/rocky-baseos-9.repo reposync -v -p repodata --download-metadata --repoid=rocky-baseos-9 --download-path=/var/www/html/rocky-baseos-9/
```

This will run the cronjob daily at midnight. Add a new line for each repository you intend to sync.

For more granular control of `cronjob` timing, please consult [this documentation on cron jobs](https://docs.rockylinux.org/guides/automation/cron_jobs_howto#more-complex-options).

### Delete old packages on sync

When running `reposync`, older packages will not be deleted by default.

This means your local repository will contain all versions of packages, as newer versions are released.

This is helpful if you desire having older packages available that are not retained in the upstream repository. However, this can take up significant storage space.

To delete packages locally that are no longer present on the remote server, append `--delete` to the `reposync` command like so:

```bash
dnf -c ./rocky-baseos-9.repo reposync -v -p repodata \
--download-metadata --repoid=rocky-baseos-9 \
--download-path=/var/www/html/rocky-baseos-9/ \
--delete
```

Although the old RPM files remain in the repository folders, they won’t be accessible via DNF on remote systems.

This is because the metadata is copied from upstream.

To retain these preexisting RPMs, update the metadata using a tool like [createrepo](https://github.com/rpm-software-management/createrepo).

## References & related articles

[How to Set Up a Web Server in Rocky Linux](https://ciq.com/blog/how-to-set-up-a-web-server-in-rocky-linux/)  
[Rocky Linux Cronjob Guide](https://docs.rockylinux.org/guides/automation/cron_jobs_howto/)  
[DNF RepoSync Options](https://dnf-plugins-core.readthedocs.io/en/latest/reposync.html)  
[DNF RepoManage Options](https://dnf-plugins-core.readthedocs.io/en/latest/repomanage.html)


How to Create a Local Mirror Using Reposync with CIQ Depot


## Introduction

This article is a great reference for customers looking to learn more about how CIQ evaluates and remediates security vulnerabilities. 

## FAQ 

**What CVE’s are addressed by CIQ Bridge and CIQ LTS?**  
For both Bridge and LTS subscriptions, CIQ focuses on resolving CVEs with CVSS scores of 7 and above. CIQ also considers customer impact and environmental factors. CIQ evaluates CVE’s, taking into account the OS, the build environment, and other Linux vendors' scores to determine the applicability of the CVE to our products. CIQ prioritizes CVE’s based on CVSS score, package exposure, popularity, confidence in the potential fixes (vs. risk of them introducing bugs), and customer requests.

**What does the CVSS Score mean?**  
CIQ evaluates scores from [NIST CVSS 3.x Scoring System](https://nvd.nist.gov/vuln-metrics/cvss), [MITRE](https://www.cve.org/) and other OS Vendors to evaluate and rank vulnerabilities in a standardized and repeatable way. These scores are used as inputs for CIQ to determine the appropriate impact of the CVE to our products.

**Are there SLA’s for CVE remediation?**  
CIQ does not provide SLA’s for any CVE remediation. They are prioritized based on the criteria discussed above.

**Where can I see the status of a given CVE?**  
All fixes can be found in the CSAF files provided by the public [CIQ Advisories Repository](https://github.com/ctrliq/advisories). CVE fixes are also published in the [CIQ Portal](https://portal.ciq.com/cve?sort=updated&direction=desc). For any other status updates, please reach out to your Customer Success Manager or open a Support ticket.

**How do CSAF files relate to the output of my scanning tool?**
Scanners use CSAF files and data from OS vendors to determine if a CVE is fixed based on the version of the RPM. Scanner vendors are responsible for processing Rocky Linux and CIQ CSAF files and advisories to determine if a fix has been released for a given CVE. If your scanner is not ingesting the CSAF files from Rocky/CIQ LTS, it will produce false positives.

**How do CVE’s relate to Security Advisories?**  
Security advisories can contain one or more CVE’s. CIQ evaluates CVEs individually and not Security Advisories. CVE’s will be fixed based on the criteria mentioned above.


CIQ CVE Remediation - Frequently Asked Questions


## Introduction

Many businesses need to have multiple desktop environments set up and to tweak them accordingly as per their requirements.

CIQ delved deep into one such test case, in order to have the MATE desktop environment installed alongside the GNOME desktop environment and to remove the ability to log in to MATE from the GNOME Display Manager (GDM) login screen.

This article will describe how to remove the option to log in to MATE from GDM.

## Problem

An administrator would like to disable the option to log in to MATE or other alternative desktops from the GDM login window without uninstalling the desktop environment.

## Resolution

Simply move the `mate.desktop` file to another location or rename it to `mate.desktop.disabled`:

```bash
mv /usr/share/xsessions/mate.desktop /usr/share/xsessions/mate.desktop.disabled
```

After a reboot, you should no longer see the option at the GDM login screen to log in to MATE when clicking the cog icon after entering your username.

## Conclusion

Following the steps listed in this article by renaming the `mate.desktop` file should help you better control the desktop environments (in this case with MATE) that users have access to from the GNOME Display Manager in your environment.

An example of where this is beneficial is with separating users into two groups: those needing an alternative desktop for specific applications and those that do not and can complete their assigned tasks using GNOME.

## References & related articles

[Rocky Linux Desktops](https://docs.rockylinux.org/desktop/)


How to Remove Access to the MATE Desktop Environment From the GNOME Desktop Manager Login Screen


## Introduction

This article provides a step-by-step guide to manually disable the screensaver lock screen setting on the GNOME desktop environment. Follow the instructions below to complete the process using `dconf` and `dconf-editor`.

## Problem

Users may need to disable the screensaver lock screen setting on their GNOME desktop environment for various reasons, such as preventing interruption during tasks.

## Prerequisites

Make sure your system includes the `dconf` and `dconf-editor` packages. If you need to install them, run the following command:

```bash
sudo dnf install -y dconf dconf-editor
```

## Instructions

### Steps to disable the screensaver lock-screen setting

1. **Launch dconf-editor**

   Open the `dconf-editor` by running the following command in your terminal:

   ```bash
   dconf-editor
   ```

   This action launches the `dconf` GUI.

2. **Navigate to the screensaver settings**

   In the `dconf-editor`, navigate to the screensaver settings by following this path:

   ```text
   org > gnome > desktop > screensaver
   ```

3. **Edit lock screen setting**

   - Click on `lock-enabled`. This opens a screen where you can change the value.
   - Toggle the `Use default value` to off.
   - Change the `Custom value` to `false`.
   - Click the blue check-mark at the bottom to confirm your changes.

   ![dconf-editor screensaver settings](/images/articles/disable-screensaver-lock-screen/screensaver-dconf-editor_m.jpg)

4. **Handling errors**

   If you encounter the following error in your terminal while attempting to save the settings:

   ```text
   (dconf-editor:17050): dconf-WARNING **: [time]: failed to commit changes to dconf: Failed to execute child process "dbus-launch" (No such file or directory)
   ```

   This indicates that the `dbus-launch` utility is missing. Resolve this by exiting the GUI and installing the `dbus-x11` package:

   ```bash
   sudo dnf install -y dbus-x11
   ```

   After installing, relaunch the GUI and attempt to save your changes again. If the error persists, reboot the system.

### Validate the changes

Verify that the lock screen is disabled by running one of the following commands:

```bash
gsettings get org.gnome.desktop.screensaver lock-enabled
```

or

```bash
dconf read /org/gnome/desktop/screensaver/lock-enabled
```

These commands should return `false`, indicating that the lock screen has been disabled.

## Conclusion

Follow these steps to manually disable the screensaver lock screen setting on your GNOME desktop environment. If you encounter any issues, verify that you have installed and correctly configured all required packages.


Disabling the Screensaver Lock Screen Setting


## Introduction

`kdump` is a kernel crash dumping mechanism that allows you to save the contents of system memory when a kernel panic or system crash occurs. This memory dump (vmcore) can be analyzed to determine the root cause of the crash. This guide covers installation, configuration, and testing of `kdump` on Rocky Linux 8, 9, and 10, including important differences between versions.

## Problem

System administrators need to capture kernel crash dumps for troubleshooting kernel panics and system failures. Without `kdump` properly configured, diagnostic information is lost when the system crashes, making root cause analysis difficult or impossible.

## Resolution

### Understanding Version Differences

Before configuring `kdump`, be aware of key differences across Rocky Linux versions:

**Rocky Linux 8:**

- Supports `crashkernel=auto` parameter for automatic memory allocation
- The `crashkernel=auto` parameter will only work when the available memory is greater than 1GB

**Rocky Linux 9:**

- The `crashkernel=auto` parameter is officially deprecated, but still available if needed

**Rocky Linux 10:**

Rocky Linux 10 follows the same crashkernel configuration approach as Rocky Linux 9. Use explicit memory range specifications for the `crashkernel` parameter, as `crashkernel=auto` remains deprecated.
### Step 1: Install `kdump` Packages

Install the required `kexec-tools` package:

```bash
sudo dnf install kexec-tools
```

Verify installation:

```bash
rpm -q kexec-tools
```

### Step 2: Configure crashkernel memory reservation

The crashkernel parameter reserves memory for the crash kernel. Configuration differs by Rocky Linux version.

#### For Rocky Linux 8

Check if crashkernel is already configured:

```bash
grep crashkernel /proc/cmdline
```

If using automatic allocation:

```bash
sudo grubby --update-kernel=ALL --args="crashkernel=auto"
```

Or specify a fixed amount:

```bash
sudo grubby --update-kernel=ALL --args="crashkernel=256M"
```

#### For Rocky Linux 9 and 10

Check current configuration:

```bash
grep crashkernel /proc/cmdline
```

Use `kdumpctl estimate` to calculate recommended memory:

```bash
sudo kdumpctl estimate
```

Example output:

```text
Reserved crashkernel:    256M
Recommended crashkernel: 192M

Kernel image size:   57M
Kernel modules size: 8M
Initramfs size:      36M
Runtime reservation: 64M
Large modules:
    xfs: 2686976
    kvm: 1404928
```

Configure the crashkernel based on the recommendation using a fixed value:

```bash
sudo grubby --update-kernel=ALL --args="crashkernel=192M"
```

Alternative: Range-based allocation (for deployments across multiple systems)

If you're deploying the same configuration to multiple servers with different RAM amounts, use a range-based allocation:

```bash
sudo grubby --update-kernel=ALL --args="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
```

The kernel automatically selects the appropriate reservation based on total system RAM at boot time:

- If system has 1-4GB total RAM: reserves 192MB for crashkernel
- If system has 4-64GB total RAM: reserves 256MB for crashkernel
- If system has 64GB+ total RAM: reserves 512MB for crashkernel

One crashkernel memory value is reserved per system - the kernel picks the matching range. This is useful for gold images, Warewulf nodes, or automated deployments where the same kernel configuration is used across different hardware.

#### Verify Bootloader Configuration

Check that the parameter was added:

```bash
sudo grubby --info=ALL | grep crashkernel
```

The `grubby` tool directly modifies the Boot Loader Specification (BLS) entries in `/boot/loader/entries/`. No additional steps are needed - the changes will take effect after reboot.

### Step 3: Configure `kdump` target and behavior

Edit the `kdump` configuration file:

```bash
sudo vi /etc/kdump.conf
```

#### Configure Dump Location

The default location is `/var/crash`. To change it, modify the `path` directive:

```bash
path /var/crash
```

For network storage (disabled by default):

```bash
#nfs [2001:db8::1:2:3:4]:/export/tmp
```

For an SSH target:

```bash
#ssh user@my.server.com
#ssh user@2001:db8::1:2:3:4
#sshkey /root/.ssh/kdump_id_rsa
```

#### Configure Core Collector

The `core_collector` allows you to modify how the `vmcore` dump is copied from the system. One feature of `core_collector` is that it includes the ability to reduce the size of the `vmcore` dump using `makedumpfile`.

```bash
core_collector makedumpfile -l --message-level 7 -d 31
```

Parameters explained:

- `-l`: Use lzo compression
- `-c`: Use zlib compression (alternative to -l)
- `-d 31`: Exclude certain memory pages to reduce dump size
- `--message-level 7`: Control verbosity of dump process

For SSH targets, add the `-F` flag:

```bash
core_collector makedumpfile -l --message-level 1 -d 31 -F
```

#### Common `kdump.conf` directives

```bash
# Default dump path
path /var/crash

# Core collector with compression
core_collector makedumpfile -l --message-level 7 -d 31

# Dump level (31 excludes free pages, cache pages, etc.)
# Valid values: 1, 2, 4, 8, 16, 31 (combination)

# For encrypted LUKS volumes, additional memory may be required
```

### Step 4: Enable and start `kdump` service

Enable `kdump` to start on boot:

```bash
sudo systemctl enable kdump
```

Start the `kdump` service:

```bash
sudo systemctl start kdump
```

Check service status:

```bash
sudo systemctl status kdump
```

You should see:

```text
● kdump.service - Crash recovery kernel arming
     Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; preset: enabled)
     Active: active (exited) since Wed 2025-11-26 15:09:01 JST; 35min ago
   Main PID: 1115 (code=exited, status=0/SUCCESS)
        CPU: 1.646s

Nov 26 15:09:00 rocky-linux96 systemd[1]: Starting Crash recovery kernel arming...
Nov 26 15:09:01 rocky-linux96 kdumpctl[1133]: kdump: kexec: loaded kdump kernel
Nov 26 15:09:01 rocky-linux96 kdumpctl[1133]: kdump: Starting kdump: [OK]
Nov 26 15:09:01 rocky-linux96 systemd[1]: Finished Crash recovery kernel arming.
```

### Step 5: Reboot to Apply Crashkernel Reservation

The crashkernel memory reservation only takes effect after a reboot:

```bash
sudo reboot
```

After reboot, verify the crash kernel is loaded:

```bash
cat /proc/cmdline | grep crashkernel
```

Check that memory has been reserved:

```bash
cat /sys/kernel/kexec_crash_size
```

This should show a non-zero value (in bytes). For example, 512MB would show as `536870912`.

Verify `kdump` is active:

```bash
sudo kdumpctl status
```

Expected output:

```text
# From Rocky Linux 9
kdump: Kdump is operational
kdump: Notice: No vmcore creation test performed!
```

### Step 6: Test kdump Configuration

Before relying on kdump in production, test that it captures crashes correctly.

**WARNING: This will crash your system. Only perform on test systems or during maintenance windows.**

Enable the kernel SysRq functionality:

```bash
echo 1 | sudo tee /proc/sys/kernel/sysrq
```

Trigger a kernel panic:

```bash
echo c | sudo tee /proc/sysrq-trigger
```

The system will crash immediately, reboot, and save the vmcore dump.

After the system restarts, check for the crash dump:

```bash
ls -lh /var/crash/
```

You should see a timestamped directory containing:

```text
vmcore           - The crash dump file
vmcore-dmesg.txt - Kernel log at time of crash
kexec-dmesg.log  - kdump kernel boot log
```

Verify the vmcore file was created and has a reasonable size:

```bash
du -sh /var/crash/*/vmcore
```

## Notes

- `kdump` requires reserved memory that is unavailable to the running system. This is a small trade-off for crash dump capability.
- For systems with encrypted LUKS volumes, additional memory may be required for `kdump` to function properly.
- The memory reservation configured with `crashkernel=` is allocated at boot time and cannot be changed without a reboot.
- Network-based crash dumps (NFS, SSH) require network connectivity during the crash kernel boot, which may not always be available.
- The dump level parameter (`-d 31`) excludes free pages, cache pages, and other non-essential memory to reduce dump size while retaining critical information.
- If `kdump` fails to start with "Memory for crashkernel is not reserved" error, verify the crashkernel parameter is set correctly and the system has been rebooted.
- On systems with Secure Boot enabled, the crash kernel must be properly signed.

## References

[Rocky Linux: Kernel Crash Dump Analysis](https://kb.ciq.com/article/rocky-linux/rl-analysis-of-vmcore-dump-with-crash)
[Rocky Linux: Kernel Panics Caused by CrowdStrike Falcon](https://kb.ciq.com/article/rocky-linux/rl-kernel-panic-crowdstrike-falcon)


How to Enable and Configure kdump on Rocky Linux


## Introduction

When installing Rocky Linux in a cloud environment, all directories, including `/tmp`, will typically be installed on a single volume. If you try to use a separate partition or volume for `/tmp`, you might encounter an issue where the entire root filesystem becomes read-only after a reboot. This occurs because the `systemd` `tmp.mount` unit is sometimes masked by default.

When a `systemd` unit is masked, it is linked to `/dev/null`, making it impossible to start. This differs from disabling a service, which prevents it from running at startup, but does allow it to be manually started or started by another process or unit.

You want the `tmp.mount` unit to be disabled so it does not start by default. However, you need `systemd` to start the `tmp.mount` service when it runs the `systemd-fstab-generator` at boot, which converts `/etc/fstab` entries into native `systemd` units. If `tmp.mount` is masked, this process fails, causing the system to mount the root directory as read-only.

## Symptoms

After you add an entry for `/tmp` in `/etc/fstab` and then reboot, you may see many errors related to a read-only filesystem. You will likely also see the following error when attempting to create a file:

```bash
touch /var/tmp/test
touch: cannot touch 'var/tmp/test': Read-only file system
```

## Resolution

Telling `systemd` to unmask this unit will allow you to mount `/tmp` according to how `/etc/fstab` is configured:

```bash
sudo systemctl unmask tmp.mount
```

If you get a read-only filesystem error when attempting this, you need to remount the root volume first:

```bash
sudo mount -o remount,rw /
```

Once the filesystem is remounted and you have verified you can write to it, rerun the `unmask` command. After a reboot, your server should start up as expected.

There is no need to enable this unit as `systemd` will dynamically generate it and start it at boot.

## Root Cause

This issue arises because cloud servers are primarily used for network-specific tasks such as web hosting.

As a result, the need for an in-memory `/tmp` directory (which is generated by default when you start `tmp.mount`), is largely negated because the network is significantly slower than the disk that `/tmp` would run on.

Additionally, since memory is a significant factor in the cost of cloud services, Rocky Linux does not rely on memory-based `tmpfs` directories. Moreover, the need for a separate partition or volume for `/tmp` is only necessary in minimal use cases and is unnecessary for the majority of workloads.

## References & related articles

freedesktop.org's page on `systemd-fstab-generator`: [Link](https://www.freedesktop.org/software/systemd/man/latest/systemd-fstab-generator.html)
Rocky Linux Admin Guide [Link](https://rocky-linux.github.io/documentation/RockyLinuxAdminGuide.pdf)


How to Avoid a Read-Only Filesystem When Using a Separate Partition or Volume for /tmp


## Introduction

This guide focuses on troubleshooting issues encountered during or after setting up rsyslog for encrypted log transport on Rocky Linux. It addresses common problems such as missing modules and misconfigured parameters that can prevent proper decryption of incoming logs. Whether you’re building a centralized logging infrastructure or enhancing your lab environment’s security, this guide provides hands-on insights into troubleshooting and correct configuration.

## Problem

When an `rsyslog` collector is configured to receive logs from multiple systems and intended to secure the log transport using encryption, errors indicating required modules are missing or misconfigured can be encountered. This can cause scenarios where logs are received on the collector, but are not being decrypted correctly, halting proper secure communication.

## Symptoms

The `rsyslog` configuration can produce several notable errors:

### Module loading issue

```bash
could not load module 'gtls', errors: trying to load module /usr/lib64/rsyslog/gtls.so: /usr/lib64/rsyslog/gtls.so: cannot open shared object file: No such file or directory
```

### Parsing errors in the configuration file

```bash
error during parsing file /etc/rsyslog.d/01-collector.conf, on or before line 9: parameter 'StreamDriver.AuthMode' not known -- typo in config file? 
error during parsing file /etc/rsyslog.d/01-collector.conf, on or before line 9: parameter 'StreamDriver.Mode' not known -- typo in config file? error during parsing file /etc/rsyslog.d/01-collector.conf, on or before line 9: parameter 'StreamDriver.Name' not known -- typo in config file?
```

Even though logs are being received, they remain encrypted and unprocessed, indicating a configuration or module issue.

## Resolution

Below are the prerequisites and detailed steps to resolve the issue:

### Testing

* Utilize the `root` account or a user with `sudo` privileges for testing.

* Set up two Rocky Linux 8.10 machines: one designated as the server and the other as the client.

* Set the hostname for each node with an FQDN:

  * Server: `hostnamectl set-hostname rocky-linux810-server.rsyslog.test`

  * Client: `hostnamectl set-hostname rocky-linux810-client.rsyslog.test`

* Update the `/etc/hosts` file on both nodes with the respective IP addresses and hostnames. An example is below (please replace the IPs and hostnames with your own):

```bash
cat << "EOF" | tee /etc/hosts
10.0.0.4    rocky-linux810-server.rsyslog.test  rocky-linux810-server
10.0.0.3    rocky-linux810-client.rsyslog.test  rocky-linux810-client
EOF
```

### Package installation

On both the server and client, install the necessary packages using the following command:
  
```bash
dnf install -y rsyslog rsyslog-gnutls rsyslog-openssl logrotate
```

### Firewall and SELinux adjustments

* Open port `6514/tcp` on both nodes:
  
```bash
firewall-cmd --add-port=6514/tcp --permanent
firewall-cmd --reload
```

* On the server, allow port `6514` through SELinux:

```bash
semanage port -m -t syslogd_port_t -p tcp 6514
```

### Certificate generation

* Create a directory for certificates:
  
```bash
mkdir -p /etc/rsyslog/keys
```

* Generate a self-signed certificate and private key:
  
```bash
openssl req -new -x509 -days 365 -nodes \
  -out /etc/rsyslog/keys/server-cert.pem \
  -keyout /etc/rsyslog/keys/server-key.pem
```

### Directory setup for log storage

* Create directories for logs on both the server and client:

```bash
# Server
mkdir -p /data/logs/rocky-linux810-server

# Client
mkdir -p /data/logs/rocky-linux810-client
```

* Set appropriate permissions for each directory:
  
```bash
# Server
chmod 644 /data/logs/rocky-linux810-server

# Client
chmod 644 /data/logs/rocky-linux810-client
```

* Configure SELinux contexts:
  
```bash
# Server
semanage fcontext -a -t var_log_t "/data/logs/rocky-linux810-server(/.*)?"
restorecon -Rv /data/logs/rocky-linux810-server

# Client
semanage fcontext -a -t var_log_t "/data/logs/rocky-linux810-client(/.*)?"
restorecon -Rv /data/logs/rocky-linux810-client
```

### Correcting the module issue

* The key resolution is to ensure the `gtls` module is correctly referenced. Since the file `/usr/lib64/rsyslog/gtls.so` module is missing, a symbolic link needs to be created:
  
```bash
ln -s /usr/lib64/rsyslog/lmnsd_gtls.so /usr/lib64/rsyslog/gtls.so
```

### Server configuration (`/etc/rsyslog.d/01-collector.conf`)

* Create/update the configuration file with the following content:
  
```bash
cat << "EOF" | tee /etc/rsyslog.d/01-collector.conf
# /etc/rsyslog.d/01-collector.conf

module(load="imtcp")
module(load="gtls")

$DefaultNetstreamDriver gtls

$DefaultNetstreamDriverCAFile   /etc/pki/tls/certs/ca-bundle.crt
$DefaultNetstreamDriverCertFile /etc/rsyslog/keys/server-cert.pem
$DefaultNetstreamDriverKeyFile  /etc/rsyslog/keys/server-key.pem

$InputTCPServerRun 6514

$template RemoteLogs,"/data/logs/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
& ~
EOF
```

### Client configuration (`/etc/rsyslog.d/01-sender.conf`)

* Create/update the configuration file on the client with:
  
```bash
cat << "EOF" | tee /etc/rsyslog.d/01-sender.conf
module(load="omfwd")
module(load="gtls")

$DefaultNetstreamDriverCAFile /etc/pki/tls/certs/ca-bundle.crt

$DefaultNetstreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon

*.* action(type="omfwd"
           target="10.0.0.4"
           port="6514"
           protocol="tcp")
EOF
```

### Service restart and testing

* Restart `rsyslog` on both the server and client:
  
```bash
systemctl restart rsyslog
```

* Test by sending an encrypted message from the client:
  
```bash
logger "THIS IS A TEST ENCRYPTED MESSAGE"
```

* On the server, verify the decrypted log messages:
  
```bash
tail -f /data/logs/rocky-linux810-client/root.log
```

* An example output:
  
```bash
tail -f /data/logs/rocky-linux810-client/root.log
[root@rocky-linux810-server ~]# tail -f /data/logs/rocky-linux810-client/root.log
Apr 30 07:26:07 rocky-linux810-client root[53589]: THIS IS A TEST ENCRYPTED MESSAGE
Apr 30 07:26:07 rocky-linux810-client root[53589]: THIS IS A TEST ENCRYPTED MESSAGE
Apr 30 07:26:08 rocky-linux810-client root[53590]: THIS IS A TEST ENCRYPTED MESSAGE
```

## Root Cause

The core issue stems from a missing symbolic link for the `gtls` module. The `rsyslog` configuration is attempting to load the module from `/usr/lib64/rsyslog/gtls.so`, which does not exist. Instead, the system provides the library as `/usr/lib64/rsyslog/lmnsd_gtls.so`. Creating the symbolic link resolves the `module-not-found` error and allows the proper configuration directives—such as `StreamDriver.Name`, `StreamDriver.Mode`, and `StreamDriver.AuthMode`—to be recognized and processed, thereby enabling successful decryption of incoming logs.

## Notes

* **SELinux and Firewall:** Always ensure that SELinux contexts and firewall rules are appropriately configured, as they can block the secure transport on non-standard ports like `6514`.

* **Module Naming and Configuration:** Module names and parameters can vary between rsyslog releases. It’s advisable to consult the [official rsyslog documentation](https://www.rsyslog.com/doc/index.html) when encountering similar issues.

* **Certificate Management:** For production environments, replace self-signed certificates with those issued by a trusted CA and ensure proper certificate management policies are in place.

## References & Related Articles

[Rsyslog Error 2066 - Missing gtls Module](https://www.rsyslog.com/?s=error+2066)  
[Rsyslog Error 2207 - Unknown Parameter Issues](https://www.rsyslog.com/?s=error+2207)  
[Rsyslog gtls Network Stream Driver](https://www.rsyslog.com/doc/concepts/ns_gtls.html)


Setting Up Rsyslog with Encrypted Transport on Rocky Linux


## Introduction

When working with multiple repositories in Rocky Linux, administrators may wish to control which packages are used from which repository.  

This is particularly useful when specific kernel versions must be preserved or when particular packages must be included from either one repository or another.  

This article outlines how to configure repository definitions to include or exclude packages and avoid unwanted package overrides.  

## Problem

Using multiple repositories with overlapping packages can lead to package conflicts or unintentional upgrades.  

For example, a FIPS-certified repository may install newer kernel packages that override those from an LTS repository if not properly managed.  

Without `exclusions` or `inclusions` in place, the `priority` of a repository alone may not prevent unwanted package selections.  

## Symptoms

Running `dnf update` attempts to install a version of a package from a repository you weren't intending to pull from.  

## Resolution

* The following example uses Rocky Linux 9.2 and the CIQ Portal `CIQ LTS for Rocky 9.2` and `CIQ FIPS 9.2 Certified` repositories, but can be applied to any installation of Rocky Linux with more than one repository.

* Install the `depot` RPM:

```bash
dnf install -y https://depot.ciq.com/public/files/depot-client/depot/depot.x86_64.rpm
```

* Log in with your CIQ Depot `Username` and `Access Token`:

```bash
depot login -u <USERNAME> -t <TOKEN>
```

* Enable the necessary repositories using `depot enable`: 

```bash
depot enable lts-9.2  
depot enable fips-9.2-certified  
```

### Excludepkgs example

* In the following example, we are going to update the `kernel*` packages. However, we want to pull and install the packages from the `CIQ LTS for Rocky 9.2` repository and not the `CIQ FIPS 9.2 Certified` repository.

* Modify the FIPS repository configuration file `/etc/yum.repos.d/depot-fips-9.2-certified.repo`.

* In the `[fips-9.2-certified-x86_64]` section, add the following line before `enabled = true`:

```ini
excludepkgs=kernel*
```

* Save the file.

* Once the changes to the file have been made, your repository configuration will look like the below example:

```ini
[fips-9.2-certified-x86_64]  
name = Rocky Linux 9.2 from CIQ - FIPS Certified (x86_64)  
baseurl = https://depot.ciq.com/files/fips-9.2-certified/fips-9.2-certified-x86_64  
gpgkey = https://ciq.com/keys/rpm-gpg-key-ciq  
username = <CIQ_PORTAL_USERNAME_HERE>  
password = <CIQ_PORTAL_PASSWORD_HERE>  
metadata_expire = 5  
priority = 20  
repo_gpgcheck = false  
gpgcheck = true  
excludepkgs=kernel*  
enabled = true  
skip_if_unavailable = true  
```

* Run `dnf update` to verify only LTS kernel packages are selected.

* Optionally use `dnf list --showduplicates` to inspect version availability across repositories. An example is with the `python3-perf` package:

```bash
python3-perf.x86_64                                  5.14.0-284.30.1.el9_2                    rocky-baseos-9.2.x86_64   
python3-perf.x86_64                                  5.14.0-284.30.1.el9_2.ciqfips.0.8.1      fips-9.2-certified-x86_64 
python3-perf.x86_64                                  5.14.0-284.30.1.el9_2.92ciq_lts.6.1      rocky-lts-9.2.x86_64      
python3-perf.x86_64                                  5.14.0-570.22.1.el9_6                    appstream
```

You can use `dnf list` variants to inspect package availability, which helps with debugging repository conflicts. Recommended commands include:

```bash
dnf list --showduplicates
dnf list --available
dnf list --installed
dnf list --obsoletes
dnf list --upgrades
```

Use these commands to understand which package versions are coming from which repositories.

### Includepkgs example

* Using the same repository setup above, `includepkgs` allows us to use particular packages from a repository and ignore all others.

* For example, you are wanting to explore and test the `openssl` package from the `CIQ FIPS 9.2 Certified` repository and don't want to use any of the other packages.

* Open the `/etc/yum.repos.d/depot-fips-9.2-certified.repo` repository.

* In the `[fips-9.2-certified-x86_64]` section, add the following line under `enabled = true`:

```ini
includepkgs=openssl,openssl-libs,openssl-fips-provider,openssl-fips-provider-so
```

* Save the file.

* Your repository configuration will look like this:

```ini
[fips-9.2-certified-x86_64]  
name = Rocky Linux 9.2 from CIQ - FIPS Certified (x86_64)  
baseurl = https://depot.ciq.com/files/fips-9.2-certified/fips-9.2-certified-x86_64  
gpgkey = https://ciq.com/keys/rpm-gpg-key-ciq  
username = <CIQ_PORTAL_USERNAME_HERE>  
password = <CIQ_PORTAL_PASSWORD_HERE>  
metadata_expire = 5  
priority = 20  
repo_gpgcheck = false  
gpgcheck = true 
includepkgs=openssl,openssl-libs,openssl-fips-provider,openssl-fips-provider-so
enabled = true  
skip_if_unavailable = true  
```

* Then when running `dnf update` and filtering by `fips`, you will find that only the above selected packages listed in the `includepkgs` line have been included and no other packages from the FIPS repository:

```bash
[root@rocky-linux92 ~]# dnf update | grep fips
 openssl                          x86_64  1:3.2.2-6.el9_2.ciqfips.1.0.1            fips-9.2-certified-x86_64   1.3 M
 openssl-libs                     x86_64  1:3.2.2-6.el9_2.ciqfips.1.0.1            fips-9.2-certified-x86_64   2.1 M
 openssl-fips-provider            x86_64  3.0.7-27.el9_2.ciqfips.0.2.7             fips-9.2-certified-x86_64   8.7 k
 openssl-fips-provider-so         x86_64  3.0.7-27.el9_2.ciqfips.0.2.7             fips-9.2-certified-x86_64   654 k
```

## Root cause

The FIPS repository has a higher `priority` value (20) than the LTS repository (50), causing it to override package versions.

Without explicitly excluding packages from the FIPS repository, DNF chooses those over the LTS versions.

By using `excludepkgs=kernel*`, DNF is instructed to ignore any kernel packages from the FIPS repository.

## References & related articles

* [dnf man page, in particular the section on dnf list](https://www.man7.org/linux/man-pages/man8/dnf.8.html)


How to Include and Exclude Packages from Repositories on Rocky Linux


## Introduction

The FIPS `openssl` version is built with strict requirements for entropy to provide the necessary security guarantees. It fills up "pools" of truly random numbers when it starts up. If these pools aren't filled, applications that call the `openssl` library will get blocked until enough random material is gathered to proceed. Regular `openssl` behaves differently. If it can't get enough true randomness, it uses fuzzy pseudorandomness and continues. It's important to understand that FIPS isn't meant to run in pieces. It's a holistic system. The kernel provides the entropy to the `openssl` library via a system call.

The RHEL/Rocky 9 kernel (and upstream 5.10, 5.15, etc. kernels) seem to provide enough entropy more or less built-in. CentOS 7 (kernel 3.10.0) and RHEL/Rocky 8 (kernel 4.18) don't in some cases.

## Problem

When using `openssl` FIPS package on a CentOS/Rocky 8 system you may see the following timeouts when using `dnf` with the FIPS `openssl` libraries installed:

```bash
dnf list
Rocky Linux BaseOS         0.0  B/s |   0  B     17:45
Errors during downloading metadata for repository 'baseos':
 - Curl error (28): Timeout was reached for https://.../BaseOS/x86_64/os/repodata/repomd.xml [SSL connection timeout]
 - Curl error (28): Timeout was reached for https://.../BaseOS/x86_64/os/repodata/repomd.xml [Operation timed out after 99165 milliseconds with 0 out of 0 bytes received]
Error: Failed to download metadata for repo 'baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
```

Also, `sshd` may freeze on startup, preventing remote access to a system.

## Resolution

Fortunately, the solution is simple:  
On any CentOS 7/Rocky 8 host, if you want to run containers with the FIPS `openssl` library installed, you need to do this (on the host, not inside the Rocky 8 container):

```bash
yum install rng-tools

systemctl enable --now rngd
```

The `rngd` daemon feeds extra, hardware generated, high-quality sources of entropy into the kernel, thus preventing the entropy pool from becoming exhausted.

## References & related articles

[man Pages - rngd](https://linux.die.net/man/8/rngd)  
[man Pages - random](https://linux.die.net/man/4/random)


Configuring Entropy on CentOS7/RockyLinux8 for FIPS OpenSSL


## Introduction

Generating a comprehensive diagnostic report using the `sosreport` utility is an essential step in troubleshooting and diagnosing issues on Rocky Linux systems.

The `sosreport` gathers critical information such as system logs, hardware details, installed software, and configurations, helping CIQ's Support Team to rapidly analyze and resolve issues.

This article covers various methods for creating an `sosreport` on Rocky Linux 8.10, Rocky Linux 9.6, and Rocky Linux 10.

## Problem

Users may encounter system instability, performance issues, hardware failures, or other problems that require detailed analysis of the entire system.

## Symptoms

Common symptoms requiring an `sosreport` include system crashes, performance degradation, unexplained errors in logs, failing applications or hardware operations, and general instability.

## Resolution

### Prerequisites

* An installation of Rocky Linux 8, 9, or 10.

* Root access or equivalent `sudo` privileges

* Install the `sos` package with `sudo dnf install -y sos` (in the above three Rocky Linux versions, the package is installed by default).

### Ways to generate an sosreport

#### Basic `sosreport` generation:

```bash
sudo sos report
```

#### Creating sosreport silently with default answers (you will not be prompted for a case ID):

```bash
sudo sos report --quiet
```

#### Generating a report in a custom location:

```bash
sudo sos report --tmp-dir=/path/to/custom/location
```

#### Generating an `sosreport` without being prompted for input:

```bash
sudo sos report --batch
```

#### Specifying a case ID or support reference:

```bash
sudo sos report --case-id=<case-number>
```

#### Limiting modules included in the report:

```bash
sudo sos report -o kernel,networking,system
```

#### Enable all available plugins when generating a report:

```bash
sudo sos report --alloptions
```

### Obfuscating an sosreport

In order to perform obfuscation of an `sosreport`, select one of the above commands and add the `--clean` flag to obfuscate the information:

```bash
sudo sos report --alloptions --clean
```

An obfuscated report would look like the following example:

```bash
sosreport-host0-8-2025-06-20-blzcdrz-obfuscated.tar.xz
```

You can even obfuscate a previously generated `sosreport` using this command:

```bash
sudo sos clean /var/tmp/<YOUR_SOS_REPORT>
```

### Encrypting an sosreport with gpg passphrase encryption

When generating an `sosreport`, run the following command to encrypt it with GPG encryption:

```bash
sudo sos report --encrypt-pass <YOUR_PASSWORD_HERE>
```

To decrypt the archive, run this command and enter the password that was set in the previous step:

```bash
sudo gpg --output /path/to/<SOSREPORT_NAME>.tar.gz --decrypt /var/tmp/<GENERATED_SOSREPORT>.tar.xz.gpg
```

### Generated sosreport location

You will find the location for all `sosreports` available under:

```bash
/var/tmp/sosreport-<HOSTNAME>-<YYYY-MM-DD>-<UNIQUE_STRING_TO_AVOID_REPORT_OVERWRITING>.tar.xz
```

## Root cause

The need for generating an `sosreport` comes from issues such as hardware incompatibility, software misconfiguration, kernel bugs, driver issues, or other system-level issues.

## Notes

Ensure adequate disk space on the target system, as `sosreports` can be large depending on the chosen modules and system state.

**_⚠️ WARNING_** _Sensitive data should be reviewed or sanitized from generated reports before being shared with any vendor._

## References & related articles

[sosreport GitHub Repository](https://github.com/sosreport/sos)


Generating sosreports on Rocky Linux


## Introduction

Google Cloud Platform Rocky Linux VMs experience hostname behavior changes where the system hostname reverts from FQDN to short name after reboot or NetworkManager restart. This issue is specific to versions running on GCP using the official `rocky-linux-8-optimized-gcp` image.

## Problem

When setting a hostname to an FQDN using `hostnamectl set-hostname test.domain.com`, the hostname correctly shows the FQDN initially. However, after a reboot or NetworkManager restart, the hostname reverts to just the short name portion.

## Symptoms

The following behavior is observed:

```bash
# hostnamectl set-hostname test.domain.com
# hostname
test.domain.com
# reboot
# hostname
test
```

NetworkManager logs show hostname changes occurring:

```text
NetworkManager: hostname: static hostname changed from "test.domain.com" to "test"
NetworkManager: policy: set-hostname: set hostname to 'test' (from system configuration)
```

Note that `hostname -f` still returns the FQDN correctly, but the basic `hostname` command only returns the short name.

## Resolution

This issue is caused by a [change](https://github.com/GoogleCloudPlatform/guest-configs/commit/01a11512fb3d674faa47a3001421c8fd72694716) in Google's guest configuration scripts. This new NetworkManager dispatch script is located at:

```text
/etc/NetworkManager/dispatcher.d/google_hostname.sh
```

The dispatcher script contains the following:

```bash
# Ensure that the hostname and IP address are set only for the primary NIC.
new_ip_address=$(jq -r .networkInterfaces[0].ip <<< $instance) new_host_name=$(jq -r .hostname <<< $instance) google_set_hostname
```

`google_set_hostname` is the script that sets the hostname to the shortened, unqualified domain name, version of the current system hostname. The NetworkManager dispatcher is a system service that executes custom scripts in response to network state changes. In this case, the script runs `if [[ $2 == "up" ]];` which runs the `google_set_hostname` any time a network interface comes online.

## References & related articles

[Google Cloud Guest Configs Repository](https://github.com/GoogleCloudPlatform/guest-configs)  
[NetworkManager Dispatcher Documentation](https://networkmanager.dev/docs/api/latest/NetworkManager-dispatcher.html)


Google Cloud Platform Changes Hostname in Rocky Linux


## Introduction

On Rocky Linux 9.4 and later, running `grub2-mkconfig` against efi may fail. A recent grub2 update changed how you should rebuild the GRUB config.

## Problem

When attempting to update the GRUB configuration by directly targeting the EFI path, the command fails indicating the file should not be overwritten.

This occurs when running:

````bash
sudo grub2-mkconfig -o /boot/efi/EFI/rocky/grub.cfg
````

The command will output the following error, and the configuration will not be updated:

````text
Running `grub2-mkconfig -o /boot/efi/EFI/rocky/grub.cfg' will overwrite the GRUB wrapper.
Please run `grub2-mkconfig -o /boot/grub2/grub.cfg' instead to update grub.cfg.
GRUB configuration file was not updated.
````

## Resolution

To correctly update the GRUB configuration on Rocky Linux 9.4 and newer, you must generate the configuration file in the `/boot/grub2/` directory instead of the `/boot/efi/EFI/rocky/` directory.

Run the following:

````bash
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
````

This will successfully generate and update the main GRUB configuration file. The EFI stub will automatically load this configuration on boot.

## Root Cause

On modern EFI systems, the file at `/boot/efi/EFI/rocky/grub.cfg` is no longer the main configuration file. Instead, it acts as a simple "stub" or "wrapper" that contains minimal code to load the actual configuration from `/boot/grub2/grub.cfg`.

This change prevents users from overwriting the critical EFI stub file, which could break the boot process, particularly on systems where that stub is signed for Secure Boot. The error message is an intentional safeguard to enforce the new, safer procedure for updating GRUB configuration.

## Related Articles

[Rocky Linux System Startup Guide](https://docs.rockylinux.org/books/admin_guide/10-boot/)


Grub2 EFI Update Error


## Introduction

EPEL [has recently switched](https://docs.fedoraproject.org/en-US/epel/branches/#_epel_10) to using separate dnf repositories and dist-git branches for all versions of Enterprise Linux 10, after the release of Enterprise Linux 10.1. [This also prompted EPEL](https://forums.rockylinux.org/t/rocky-9-7-rocky-10-1-and-epel/19879) for Enterprise Linux 9 to be moved to EPEL 9.7

One impact of this change, is that some packages such as `gdal-devel` were incorporated directly into a point release and made no longer available in older minor versions of EPEL. Users on lower minor releases of Enterprise Linux (Rocky Linux 9.6 for example), were therefore unable to pull certain packages that they needed from EPEL.

## Problem

Users on CIQ's Rocky Linux LTS product line are unable to acquire the packages that they need from EPEL.

## Symptoms

For example, you are running Rocky Linux 9.6 and have the EPEL repository enabled. You attempt to install the `liblas-devel` package and observe the following error:

```console
[root@rocky-linux96 ~]# dnf install liblas-devel
# Output truncated
Error: 
 Problem: conflicting requests
  - nothing provides gdal-devel needed by liblas-devel-1.8.2-0.3.gitded4637.el9.x86_64 from epel
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
```

## Resolution

To keep getting access to EPEL packages, please utilize the EPEL archive by performing the following steps.

Modify `/etc/yum.repos.d/epel.repo` and point the `epel` repository to the EPEL archive. The below example fixes the above error as shown in the `Symptoms` section:

```bash
sed -i 's|#baseurl=https://download.example/pub/epel/.*/Everything/$basearch/|baseurl=https://dl.fedoraproject.org/pub/archive/epel/9.6/Everything/$basearch/|' /etc/yum.repos.d/epel.repo
sed -i '/^\[epel\]$/,/^\[/{s|^metalink=|#metalink=|}' /etc/yum.repos.d/epel.repo```
```

Observe that the `liblas-devel` package can now be installed successfully:

```console
[root@rocky-linux96 ~]# dnf install liblas-devel
# Output truncated
Installing:
liblas-devel x86_64 1.8.2-0.3.gitded4637.el9 epel 112 k
```

## Root Cause

Some EPEL packages being removed from EPEL and then incorporated into later Enterprise Linux minor version releases, breaks packages on older Enterprise Linux minor versions that are reliant on EPEL.
# Notes
- Pinning your EPEL repository to a specific archive version will prevent you from receiving security updates and bug fixes for EPEL packages.

## References & related articles

[EPEL Branches - Fedora Docs](https://docs.fedoraproject.org/en-US/epel/branches/#_epel_10)  
[Rocky 9.7, Rocky 10.1 and EPEL - Rocky Linux Forum](https://forums.rockylinux.org/t/rocky-9-7-rocky-10-1-and-epel/19879)



How to Access the EPEL Archive on Rocky Linux


## Introduction

ICMP (Internet Control Message Protocol) is a fundamental part of networking, commonly used for diagnostic tools like `ping`. It is a set of rules used by devices to report data transmission errors within a network. During message exchanges between a sender and receiver, unexpected errors may occur, such as messages being too long or data packets arriving out of order, preventing proper assembly. In these instances, the receiver uses ICMP to notify the sender of the error and requests the message to be resent.

## Problem

Limiting the rate of ICMP echo replies is an important security measure to mitigate the risk of ICMP-based attacks, such as ICMP flood attacks. While the `tc` (Traffic Control) command is commonly used to manage traffic flow on Linux, including rate-limiting ICMP replies, an alternative approach is to use iptables through Firewalld's [direct](https://firewalld.org/documentation/direct/) rules.

## Resolution

### Create a new rule

The appropriate limits depend on your security policy and environment, but a good starting point is a rate of 1 reply per second with a burst limit of 5. This allows the server to respond to up to 5 pings in quick succession before enforcing the 1-per-second restriction. The burst setting also helps accommodate short spikes in legitimate traffic:

```shell
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 10 -p icmp --icmp-type echo-request -j DROP
firewall-cmd --reload
```

The `0` and `10` in these rules specify their priority within firewalld. Firewalld processes rules in ascending order of priority, so the first rule with priority 0 will be evaluated before the second rule with priority 10. If traffic matches the first rule meaning its within the rated limit, it will be accepted and will not proceed to the second rule. If the traffic does not match the first rule meaning it exceeds the rate limit, it will advance to the second rule and will be dropped.

### Replace the existing rule

If you need to modify this rule, you will need to delete the rule and recreate it. To see your current list of rules:

```shell
[root@rockylinux ~]# firewall-cmd --direct --get-all-rules

ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
ipv4 filter INPUT 10 -p icmp --icmp-type echo-request -j DROP
```

Now, append the rule you want to delete to `firewall-cmd --direct --permanent --remove-rule`. When adding a new rule, ensure that your priority is set lower than the priority of the `DROP` rule. In our case we will keep with the same priority of `0`:

```shell
firewall-cmd --direct --permanent --remove-rule ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 2/sec --limit-burst 7 -j ACCEPT
firewall-cmd --reload
```

### Delete the rules

To remove these rules completely, we can use the same `--remove-rule` flag as mentioned above. First start by viewing the existing list of rules as before:

```shell
[root@rockylinux ~]# firewall-cmd --direct --get-all-rules

ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
ipv4 filter INPUT 10 -p icmp --icmp-type echo-request -j DROP
```

Now, append the rules you want to delete to `firewall-cmd --direct --permanent --remove-rule`:

```shell
firewall-cmd --direct --permanent --remove-rule ipv4 filter INPUT 0 -p icmp --icmp-type echo-request -m limit --limit 1/sec --limit-burst 5 -j ACCEPT
firewall-cmd --direct --permanent --remove-rule ipv4 filter INPUT 10 -p icmp --icmp-type echo-request -j DROP
firewall-cmd --reload
```

## Notes

If you need to create, modify, or delete these rules in an environment where the Firewalld service cannot be started (such as a chrooted environment or in a kickstart), you can use [firewall-offline-cmd](https://firewalld.org/documentation/man-pages/firewall-offline-cmd.html) to modify them.

## References & related articles

[Rocky Linux Beginners Guide to Firewalld](https://docs.rockylinux.org/guides/security/firewalld-beginners/)  
[Firewalld Official Documentation](https://firewalld.org/documentation/)  
[Firewalld Direct Rules Documentation](https://firewalld.org/documentation/direct/)  
[Firewalld Offline Command Documentation](https://firewalld.org/documentation/man-pages/firewall-offline-cmd.html)


Configuring ICMP Rate Limiting in Firewalld


## Introduction

It is inevitable that hardware will fail and there are hundreds of possible scenarios.

This guide will detail troubleshooting steps which can be used to identify hardware errors and how to rectify them.

## Problem

A customer informed CIQ that a node in their production cluster had spontaneously crashed and then rebooted itself.

## Symptoms

Checking `/var/log/messages` from the `sosreport`, there were hundreds of `[Hardware Error]: Hardware error from APEI Generic Hardware Error Source` messages observed.

## Resolution

### Prerequisites

Access to the `root` user or a user with escalated privileges.

### Creating an sosreport

When creating a ticket with CIQ, generating and attaching an `sosreport` to the ticket is very helpful. This allows the support team to start their investigation, without further having to ask the customer to generate one.

Install the `sos` package:

```bash
dnf install -y sos
```

Run the `sos report --batch` command to perform unattended data gather of your system.

Once complete, you will find the `sosreport` under `/var/tmp/sosreport-hostname-yyyy-mm-dd-abcdefg.tar.xz`. An example is below:

```bash
[root@Rocky-Linux-9-5-Test-Machine ~]# ls -l /var/tmp/ | grep sosreport
-rw-------. 1 root root 11094320 Dec 27 02:33 sosreport-Rocky-Linux-9-5-Test-Machine-2024-12-27-elpmjjq.tar.xz
-rw-r--r--. 1 root root       65 Dec 27 02:33 sosreport-Rocky-Linux-9-5-Test-Machine-2024-12-27-elpmjjq.tar.xz.sha256
```

Please upload the `sosreport` to the ticket.

### Analysis of the issue

When observing behavior such as crashes or random reboots, the best log source is `/var/log/messages` - this records the global messages that are generated from the various services running on your server.

Hardware errors are usually identified by the kernel in the `/var/log/messages` log with a `[Hardware Error]` prefix. These messages come from the ACPI Platform Error Interface.

Using a combination of tools such as `grep`, `awk`, and `uniq` you can quickly find and triage repeating errors. The following in `/var/log/messages` from a customer's `sosreport` shows hundreds of memory-related errors:

```bash
grep "Hardware Error" ./sosreport-<NODE_NAME>-2024-12-24-gwwbeqs/var/log/messages | awk '{print $8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20}' | grep "memory error" | uniq -c
 710 section_type: memory error
```

In addition to when a `Hardware Error` occurs, you will also encounter messages of `Corrected error, no action required`. While the error may be temporarily corrected, if these messages continue, it is a sign of a more serious fault. In that scenario, it is best to contact the hardware vendor and replace the affected part outright.

### Recommendations

In the above memory fault example, the recommendation to the customer was to change out their memory modules and they then contacted their hardware team for further analysis.

## References & related articles

`awk` man page: https://man7.org/linux/man-pages/man1/awk.1p.html

`grep` man page: https://man7.org/linux/man-pages/man1/grep.1.html

`sosreport` man page: https://linux.die.net/man/1/sosreport

`uniq` man page: https://man7.org/linux/man-pages/man1/uniq.1.html


How to Identify Hardware Faults in Rocky Linux


## Introduction

A quick intro to the topic at hand.

In some technical processes, reading from and writing to registers may lead to inconsistencies or invalid values due to the simultaneous execution of read and write operations. This article provides an overview of this issue and offers potential solutions for troubleshooting it.

## Problem

If gathering the PCI information from /sys (sysfs), the data shows PCI link speed and status. In some cases, the LnkSta that is returned indicates the following "16GT/s(downgraded)  x4(downgraded)". While 16GT/s and x4 might be the correct speed and width, the output shows "downgraded".

```console
$ lspci | grep downgr
...
        LnkSta: Speed 8GT/s (downgraded), Width x8 (ok)
        LnkSta: Speed 16GT/s (ok), Width x8 (downgraded)
...
```

### This could match the following criteria

- Inconsistencies in data when comparing multiple reads from the same register
- Observing incorrect or unexpected values that do not align with the expected behavior.
- Frequent occurrences of invalid values (e.g., once or twice a week).

## Resolution

While this is not a resolution to the problem, this could help gathering information that you can bring to your hardware provider.

Below is the lspci code that emits the string: “(downgraded)”. Just so we know what criteria is being used here:

```c
static char *link_compare(int type, int sta, int cap)
{
  if (sta > cap)
    return " (overdriven)";
  if (sta == cap)
    return "";
  if ((type == PCI_EXP_TYPE_ROOT_PORT) || (type == PCI_EXP_TYPE_DOWNSTREAM) ||
      (type == PCI_EXP_TYPE_PCIE_BRIDGE))
    return "";
  return " (downgraded)";
}
```

This is called from: cap_express_link()

```c
  w = get_conf_word(d, where + PCI_EXP_LNKSTA);
  sta_speed = w & PCI_EXP_LNKSTA_SPEED;
  sta_width = (w & PCI_EXP_LNKSTA_WIDTH) >> 4;
  printf("\t\tLnkSta:\tSpeed %s%s, Width x%d%s\n",
        link_speed(sta_speed),
        link_compare(type, sta_speed, cap_speed),
        sta_width,
        link_compare(type, sta_width, cap_width));
```

The `link_compare()` code comes from [this discussion:](https://lore.kernel.org/all/20210318171122.GA151712@bjorn-Precision-5520/T/)

In sum, it’s checking the status speed from `PCI_EXP_LNKSTA_SPEED` and if it’s less than cap_speed then it’s always reporting “(downgraded)” for devices other than `PCI_EXP_TYPE_ROOT_PORT`, `PCI_EXP_TYPE_DOWNSTREAM`, `PCI_EXP_TYPE_PCIE_BRIDGE`.

The PCI configuration data, including speed and width information that `lspci` code reads and potentially reports as “downgraded,” is pulled from the [sysfs](https://docs.kernel.org/PCI/sysfs-pci.html) entries under `/sys/bus/pci/devices`. As an example, for NVMe devices, the data is accessed from the corresponding entries for each NVMe device under `/sys/bus/pci/devices/[DEVICE_ID]/config`.

The reading of PCI configuration data is managed by the kernel's PCI subsystem, specifically within the PCI sysfs interface and related functions. The primary functions involved include:

```text
pci_read_config() in drivers/pci/pci-sysfs.c
pci_user_read_config_##size in drivers/pci/access.c
```

These functions handle the reading of configuration data directly from the hardware registers of the PCI devices.

If you use a script that reads these data, is good to understad that it is just getting raw data out of the device on the PCI bus.

For debugging you could try using dmesg and other system logs to check for any hardware events or errors reported by the kernel that might indicate link renegotiation or other PCIe issues.

Look for AER (Advanced Error Reporting) messages which can sometimes provide clues about PCIe link issues:

```bash
dmesg | grep AER
```

## Possible Causes

1. Simultaneous read and write operations on the same register: This can result in capturing an intermediate state instead of the final value, leading to inconsistencies.
2. Incorrect timing between read and write operations: If there's not enough time between reading and writing, it may lead to the capture of an incorrect or invalid value.
3. Hardware malfunctions: Faulty hardware components or software glitches can cause unexpected behavior in register values.

## References & related articles

[lspci: Don't report PCIe link downgrades for downstream ports](https://lore.kernel.org/all/20210318171122.GA151712@bjorn-Precision-5520/T/)  
[Accessing PCI device resources through sysfs](https://docs.kernel.org/PCI/sysfs-pci.html)


Inconsistent Values When Reading and Writing PCI Registers


## Introduction

CIQ SIG/Cloud Next is a set of repositories designed to accelerate new cloud features for Rocky Linux, as well as to provide access to proprietary packages that cannot be included in the official Rocky SIG/Cloud due to licensing restrictions. There are currently two repositories: an open-source repository that adds new features, and a Nonfree repository for proprietary software such as drivers for NVIDIA. CIQ SIG/Cloud Next is not intended to replace Rocky SIG/Cloud, but rather to supplement it with proprietary components. You can find the official repository and documentation for the Cloud Next repository on [gitlab](https://gitlab.com/ctrl-iq-public/sig-cloud-next).

## Instructions

We first need to install the appropriate repositories:

### Rocky 9

To install CIQ SIG/Cloud Next on Rocky 9, run the following:

```shell
sudo dnf install https://depot.ciq.com/public/download/ciq-sigcloud-next-9/ciq-sigcloud-next-9.x86_64/Packages/c/ciq-sigcloud-next-release-6-1.el9_5.cld_next.noarch.rpm
```

You can also install CIQ SIG/Cloud Next - Nonfree by running:

```shell
sudo dnf install https://depot.ciq.com/public/download/ciq-sigcloud-next-9/ciq-sigcloud-next-9-nonfree.x86_64/Packages/c/ciq-sigcloud-next-nonfree-release-3-1.el9_5.cld_next.noarch.rpm
```

### Rocky 8

To install CIQ SIG/Cloud Next on Rocky 8, run:

```shell
sudo dnf install https://depot.ciq.com/public/download/ciq-sigcloud-next-8/ciq-sigcloud-next-8.x86_64/Packages/c/ciq-sigcloud-next-release-6-1.el8_10.cld_next.noarch.rpm
```

To install the CIQ SIG/Cloud Next - Nonfree repository, run:

```shell
sudo dnf install https://depot.ciq.com/public/download/ciq-sigcloud-next-8/ciq-sigcloud-next-8-nonfree.x86_64/Packages/c/ciq-sigcloud-next-nonfree-release-3-1.el8_10.cld_next.noarch.rpm
```

### Update the system

Once you have the appropriate repositories installed, you can update your kernel by running `dnf update` or `dnf update kernel`.

### Install Nvidia drivers

You can now also install the NVIDIA Data Center GPU drivers with:

```shell
sudo dnf install nvidia-dc-driver-latest kmod-nvidia-dc-open-latest
```

If you need to choose a specific version of the driver (such as 550 or 570), you can install those directly:

```shell
sudo dnf install nvidia-dc-driver550 kmod-nvidia-dc-open550 --allowerasing
```

This will switch you to the 550 major version, and will not be updated to another major version of the package.

### Mellanox drivers

To make use of the NVIDIA Mellanox DOCA drivers, you'll need to enable the upstream DOCA repo. You can do so by running:

```shell
sudo dnf install doca-repo
```

Once this new repository is installed, you can then install the DOCA packages, including the kernel modules built for Rocky Linux, by installing the following:

```shell
sudo dnf install doca-ofed
```

### Reboot

Once you have the kernel and desired drivers installed, reboot your server.

## Notes

If you already have a kernel installed that is newer than the latest available kernel from CIQ SIG/Cloud Next, you will need to specify which version you would like to install. We first need to look for the available kernels:

```shell
$ sudo dnf --disablerepo="*" --enablerepo="ciq-sigcloud-next" list --showduplicates kernel
Installed Packages
kernel.x86_64       5.14.0-503.14.1.el9_5       @baseos
kernel.x86_64       5.14.0-503.38.1.el9_5       @baseos
Available Packages
kernel.x86_64       5.14.0-503.22.1.el9_5.cld_next.3.1        ciq-sigcloud-next
kernel.x86_64       5.14.0-503.35.1.el9_5.cld_next.2.1        ciq-sigcloud-next
```

In this case we want to install `5.14.0-503.35.1.el9_5.cld_next.2.1`, so we can run the following:

```shell
sudo dnf install kernel-5.14.0-503.35.1.el9_5.cld_next.2.1
```

Once this finishes installing, it should automatically become your default kernel at boot. You can verify this with:

```shell
$ sudo grubby --default-title
Rocky Linux (5.14.0-503.35.1.el9_5.cld_next.2.1.x86_64) 9.5 (Blue Onyx)
```

If this is the incorrect kernel, or you need to rollback to another kernel, you can do so by running the following:

```shell
$ grubby --set-default /boot/vmlinuz-5.14.0-503.35.1.el9_5.cld_next.2.1.x86_64
The default is /boot/loader/entries/0a7500f2dd6247d69028b90889d6d0fb-5.14.0-503.35.1.el9_5.cld_next.2.1.x86_64.conf with index 3 and kernel /boot/vmlinuz-5.14.0-503.35.1.el9_5.cld_next.2.1.x86_64
```

## References & related articles

[CIQ SIG/Cloud Next GitLab Repository](https://gitlab.com/ctrl-iq-public/sig-cloud-next)  
[Rocky Linux SIG/Cloud Documentation](https://sig-cloud.rocky.page)  
[Rocky Linux Kernel Guide](https://docs.rockylinux.org/labs/systems_administration_II/lab7-the_linux_kernel/)


Install the Cloud-Next Kernel in Rocky Linux


## Introduction

Production Rocky Linux systems running CrowdStrike Falcon Sensor may experience kernel panics with general protection faults during container operations. The crashes occur in `__kmem_cache_alloc_node` due to memory corruption originating from the `falcon_lsm_serviceable` kernel module. This article addresses the specific crash signature, diagnostic indicators, and resolution steps for Falcon-induced kernel panics on Rocky Linux 9.

## Problem

Servers running CrowdStrike Falcon Sensor experience kernel panics during container runtime operations, particularly when executing binaries via `runc` or similar container tools. The crash occurs in the kernel's memory allocator (`__kmem_cache_alloc_node`) when Falcon's security hooks intercept system calls. Crash analysis shows the fault originates in Falcon's kernel modules.

## Symptoms

The crash signature is highly specific and identifiable in crash dumps and dmesg logs:

**Primary indicators:**

- General protection fault with non-canonical memory addresses such as `0x289d17010b1fa10a` or `0x60c5387b1ee398de`
- Crash occurs in `__kmem_cache_alloc_node+0x10e/0x2e0` or `__kmalloc`
- Process triggering crash: `runc`, `containerd`, or other container execution tools
- Kernel tainted with `P` (proprietary) and `E` (unsigned module) flags
- Multiple stack frames from Falcon modules: `falcon_lsm_serviceable`, `falcon_lsm_pinned_*`
- Specific Falcon functions in call stack:
  - `cshook_systemcalltable_pre_compat_sys_ioctl`
  - `cshook_security_bprm_check_security`
  - `pinnedhook_security_bprm_check_security`
  - `_ZdlPvmSt11align_val_t` (C++ memory management in kernel space)

**Example crash signature from dmesg:**

```text
general protection fault, probably for non-canonical address 0x289d17010b1fa10a: 0000 [#1] PREEMPT SMP NOPTI
CPU: 9 PID: 3903395 Comm: runc Kdump: loaded Tainted: P E ------- --- 5.14.0-503.33.1.el9_5.x86_64 #1
RIP: 0010:__kmem_cache_alloc_node+0x10e/0x2e0
```

## Resolution

### Step 1: Confirm Falcon as root cause

Extract and examine the crash dump or dmesg output. Look for the specific Falcon crash signature in the call stack:

```text
Call Trace:
 __kmem_cache_alloc_node+0x10e/0x2e0
 __kmalloc+0x4b/0x140
 cshook_systemcalltable_pre_compat_sys_ioctl+0x3b505/0x43340 [falcon_lsm_serviceable]
 cshook_security_bprm_check_security+0x57/0x70 [falcon_lsm_serviceable]
 pinnedhook_security_bprm_check_security+0x39/0x80 [falcon_lsm_pinned_18108]
 security_bprm_check+0x7b/0x100
 do_execveat_common.isra.0+0x1a4/0x280
 __x64_sys_execve+0x3f/0x50
 do_syscall_64+0x5c/0x80
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
```

**Key diagnostic observation:** in typical crashes, Falcon modules account for a significant portion of the call stack such as 23 out of 50 frames, while hardware components like `MegaSAS` and `bnxt_en` show zero appearances in the execution path and no errors in `dmesg`.

To examine crash information from kdump:

```bash
# Check the auto-generated dmesg file
grep -A 30 "Call Trace" /var/crash/*/vmcore-dmesg.txt | grep falcon
```

### Step 2: Rule out hardware components

Verify that hardware components aren't contributing to the crashes:

```bash
# Verify the RAID controller shows no errors
sudo dmesg | grep -i "mega\|raid" | grep -i error

# Verify the network driver shows no errors
sudo dmesg | grep -i "bnxt\|network" | grep -i error

# Check if these components appear in the crash stack
grep -E "megaraid|bnxt_en|mpt3sas" vmcore-dmesg.txt
```

If the hardware components are clean and absent from the crash stack, the issue is isolated to Falcon.

### Step 3: Contact CrowdStrike support

Engage CrowdStrike support with the crash analysis showing Falcon module involvement:

- Provide the `vmcore` dump or extracted `dmesg` logs showing the call stack
- Include the Falcon Sensor version: `sudo /opt/CrowdStrike/falconctl -g --version`
- Include the kernel version: `uname -r`
- Reference the specific functions in the crash: `cshook_systemcalltable_pre_compat_sys_ioctl`, `falcon_lsm_serviceable`
- Request information about known issues with your kernel version and container runtimes

### Step 4: Temporarily disable Falcon for isolation test

On a non-production or isolated affected system, turn off Falcon to definitively confirm if that is the root cause or not:

```bash
# Stop Falcon sensor (this should unload modules automatically)
sudo systemctl stop falcon-sensor

# Disable automatic startup
sudo systemctl disable falcon-sensor

# Verify Falcon modules are unloaded
lsmod | grep falcon

# If modules remain loaded, unload them manually
for module in $(lsmod | grep falcon | awk '{print $1}'); do
    sudo modprobe -r "$module"
done
```

Monitor the system under normal container workload. If kernel panics cease without Falcon loaded, this definitively confirms Falcon as the root cause.

### Step 5: Apply resolution from CrowdStrike

Follow the resolution provided by CrowdStrike support. This may include Sensor updates, configuration changes, or kernel version recommendations based on their compatibility matrix.

### Step 6: Re-enable and monitor

After applying updates, re-enable Falcon and monitor for any recurrence:

```bash
sudo systemctl enable falcon-sensor
sudo systemctl start falcon-sensor

# Verify Falcon modules loaded
lsmod | grep falcon

# Monitor for crashes
sudo journalctl -k -f | grep -i "general protection\|kernel panic"
```

## Notes

- This specific crash pattern occurs when Falcon sensor intercepts container execution (`execve` system calls) during binary launches.
- The crash occurs when Falcon's `cshook_systemcalltable_pre_compat_sys_ioctl` function attempts memory allocation and encounters corrupted pointers.
- Diagnostic evidence includes stack traces showing Falcon module involvement and absence of hardware errors in `dmesg` logs.
- The non-canonical memory addresses indicate pointer corruption within the Falcon module code path.
- This issue affects containerized workloads more frequently than traditional process execution due to the higher rate of `execve` calls in container environments.
- Falcon sensor versions and kernel compatibility should be verified together - not all sensor versions are certified for all kernel versions.

## References & Related Articles

[Rocky Linux: How to Enable and Configure kdump](https://kb.ciq.com/article/rocky-linux/rl-enable-configure-kdump)
[Rocky Linux: Kernel Crash Dump Analysis](https://kb.ciq.com/article/rocky-linux/rl-analysis-of-vmcore-dump-with-crash)


Kernel Panics Caused by CrowdStrike Falcon Sensor on Rocky Linux


## Introduction

This article provides guidance on migrating a Rocky Linux virtual machine created locally using KVM to Azure.

Follow these steps to ensure a smooth migration from your local Rocky Linux KVM-generated VM to Azure.

## Problem

You need to migrate a Rocky Linux virtual machine created in a local KVM environment over to Azure.

## Symptoms

You have a Rocky Linux VM running locally on KVM and require it to run in Azure.

## Resolution

### Prerequisites

#### KVM

* Rocky Linux host with at least a 500GB disk, 32GB of RAM, and 16 vCPUs available.

* `libvirt` installed (please see [Setting up libvirt on Rocky Linux](https://docs.rockylinux.org/guides/virtualization/libvirt-rocky/) on how to install `libvirt` on Rocky Linux.

* Rocky Linux DVD ISO.

#### Azure

* Azure subscription.

* Azure Recovery Services vault.

* Azure Virtual Network.

### Steps to Resolution

#### Azure step 1

In Azure, navigate to `Recovery Services vaults` and create a new vault under your `Resource Group`:

![The "Create Recovery Services vault" "Basics" step from Azure is shown.](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-1.webp)

Then navigate to `Virtual networks`, select `Create`, and select your `Resource Group` and your `Region`.

!["Create virtual network" "Basics" step is displayed.](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-2.webp)

Under your `Recovery Services vault`, select `Enable Site Recovery`, then `VMware machines to Azure`, and after that click `1: Prepare Infrastructure`.

![In the example "Recovery Services Vault", VMware machines to Azure is highlighted and then the link to "1: Prepare infrastructure" is also highlighted beneath that.](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-3.webp)

* Download the `.OVA` image from Azure to your local machine.

#### KVM steps

Set up a new Rocky Linux VM on KVM. Assign it a 16GB disk, at least 2 vCPUs, and a minimum of 4096 MB RAM. `virt-manager` is the recommended option.

Before initiating the installation of Rocky Linux, under `Hardware` settings, change the `Disk bus` for device `VirtIO Disk 1` from `VirtIO` to `SCSI`. This is needed in order for the disk to boot on Azure once the VM is migrated over.

![The Hardware settings of a test Rocky Linux VM in virt-manager is displayed, with "VirtIO Disk 1" selected and the "Disk bus" of "SCSI" chosen in the drop-down menu in the middle of the screen.](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-4.webp)

Apply the changes and install Rocky Linux.

Ensure all necessary firewall ports are open:

```bash
sudo firewall-cmd --add-port=443/tcp --permanent
sudo firewall-cmd --add-port=1024-65535/tcp --permanent
sudo firewall-cmd --add-port=135/tcp --permanent
sudo firewall-cmd --add-port=443/udp --permanent
sudo firewall-cmd --reload
```

After installing Rocky Linux, shut down the VM and remove the `SATA CDROM 1` from VM's hardware list.

![Hardware list in virt-manager for the Rocky Linux VM, highlighting that "SATA CDROM 1" needs to be removed. The "SATA CDROM 1" option is under the "SCSI Disk" option on the left-hand side](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-5.webp)

In the VM's hardware list, select the XML tab (you need to enable XML editing in `virt-manager` beforehand) and modify the XML configuration of the `SCSI Disk 1` exactly as in the following:

```xml
<disk type="file" device="disk">
  <driver name="qemu" type="qcow2"/>
  <source file="/PATH/TO/<YOUR_QCOW2_IMAGE>.qcow2"/>
  <target dev="sda" bus="scsi"/>
  <address type="drive" controller="0" bus="0" target="0" unit="0"/>
</disk>
```

Power up the VM and verify it boots successfully.

#### Azure Site Recovery VM steps

Extract the required `vmdk` image files from the `.ova` image that you downloaded earlier in the Azure steps:

```bash
tar -xvf ./MicrosoftDRAppliance.ova
```

Convert the extracted `.vmdk` images to `.qcow2`:

```bash
qemu-img convert -O qcow2 MicrosoftDRAppliance_disk0.vmdk MicrosoftDRAppliance_disk0.qcow2
qemu-img convert -O qcow2 MicrosoftDRAppliance_disk1.vmdk MicrosoftDRAppliance_disk1.qcow2
```

Create a new VM in `virt-manager` using the `MicrosoftDRAppliance_disk0.qcow2` image, set the OS as `Windows Server 2016`, provision the VM with a minimum of 16GB RAM, 8 vCPUs, and 200GB storage. 

Before starting the VM, attach the `MicrosoftDRAppliance_disk1.qcow2` image as a separate disk.

![Adding the "MicrosoftDRAppliance_disk1.qcow2" image to the Azure Site Recovery VM under "Storage" --> "Manage" and then clicking "Finish".  ](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-6.webp)

Boot the VM and complete the initial setup, sign-in to Azure and provide your `Registration key` that you received from the `1: Prepare Infrastructure` step in Azure.

Under `Configure vCenter details` and `Provide physical server details`, make sure you select the options for a Physical server being migrated and provide the `root` login credentials and IP address for your local Rocky Linux VM.

The process will take about 30 minutes to complete, with a green checkmark shown at the end if it was successful or not.

#### Azure step 2

In Azure Portal, navigate to `Recovery Services vault`, then `Site Recovery`, after that `VMware machines to Azure` and then `Enable replication`.

Under `Machine type`, select `Physical machines`, choose your Rocky Linux VM and click `Next`.

Under `Source settings`, leave everything as default, and set a `Target name` for your Rocky Linux VM. An example is `rocky-linux-kvm-vm`. Once done, click `Next`.

Under `Target properties`, select your `Failover Azure network` and `Cache storage account`, keep everything else as default, and click `Next`.

Click `Enable replication`.

Check the replication status under `Replicated items` in your `Recovery Services vault`.

["Replicated items" in Azure, displaying the Rocky Linux VM in "Enabling protection" "Status" with a "Replication Health" of "Healthy"](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-7.webp)

Once the status of `Enabling protection` has gone, you can then initiate a `Test Failover`.

["Replicated items" highlighting the "Test Failover" option that can be selected by clicking the menu with the three dots on the right-hand side of the screen.](/images/articles/rocky-linux/rl-kvm-vm-to-azure/rl-kvm-vm-to-azure-8.webp)

Once replication completes, initiate a `Test Failover`.

After successful testing, verify the VM is available and running under `Virtual machines` in Azure.

Select `Cleanup test failover` under `Replicated items` to remove the VM that was generated.

Power down your local KVM Rocky Linux VM.

Under `Replicated items`, choose `Failover`.

Once the `Failover` is complete, go back to your `Replicated items` page and click `Complete Migration`.

If the new Virtual Machine in Azure is working as expected, then press `OK` and Azure will then cleanup all of the configurations and settings that were generated during the Migration process.

Your local KVM VM is now up and working on Azure!

## Root cause

Migration of a KVM-based Rocky Linux VM to Azure requires specific configurations, including correct disk types (SCSI), appropriate firewall configurations, and usage of the Azure Site Recovery Appliance for replication and failover.

## Notes

Ensure proper firewall and disk configuration before migration to prevent connectivity and boot issues.

Always validate the VM in Azure using the test failover feature before committing to a full failover.

## References & related articles

[Azure Site Recovery Overview](https://learn.microsoft.com/en-us/azure/site-recovery/site-recovery-overview)

[Azure Documentation](https://learn.microsoft.com/en-us/azure/?product=popular)


Migrate a Rocky Linux KVM Virtual Machine to Azure


## Introduction

This article provides step-by-step instructions for migrating a local `Rocky Linux 8.10` `KVM` virtual machine to `Microsoft Azure` using `Azure Migrate` and the `Azure Site Recovery` replication appliance.

The migration process involves preparing your source `VM`, setting up the necessary `Azure` infrastructure, configuring the replication appliance, and performing the actual migration.

## Problem

Organizations need to migrate on-premises virtual machines often running on `KVM` to `Microsoft Azure` for cloud adoption, disaster recovery, or infrastructure modernization purposes. The amount of VMs to migrate to the cloud can range in the thousands and services such as `Azure Migrate` help to make the transition smoother.

The migration process requires specific configuration steps for both the source `VM` and `Azure` environment to ensure successful replication and migration.

## Symptoms

You have a fleet of `Rocky Linux` `VM`s running on `KVM` that needs to be migrated to `Azure`.

The `VM` requires migrating to the cloud whilst maintaining its configuration and data integrity.

## Resolution

### Prerequisites

#### Rocky Linux

* A `Rocky Linux 8.10` `KVM` `VM` (as an example, but Rocky Linux 9 and Rocky Linux 10 are also fine) with a minimum of:

* 4096 MB of `RAM`

* 2 `vCPUs`

* 16GB of disk space

* `SCSI` disk type

#### Windows Server

* A `Windows Server 2022` `VM` on `KVM`, also with a minimum of:

* 16GB of `RAM`

* 8 `vCPUs`

* 700GB of disk space (the replication appliance requires at least 600GB of disk space cache available)

*Please check [Azure Appliance requirements](https://learn.microsoft.com/en-us/azure/migrate/migrate-replication-appliance?view=migrate-classic#appliance-requirements)*

#### Azure

* An active `Azure` subscription with appropriate permissions

* Access to the `Azure Portal`

### Step 1: prepare the source Rocky Linux VM

* Deploy a local `Rocky Linux 8.10` `KVM` and upgrade all packages.

* Provide the `VM` with 4096 MB of `RAM`, 2 `vCPUs` and 16GB of disk as listed in the Prerequisites.

* Change the bus type to `SCSI` before installing the `OS`. This is performed under `Hardware` --> `VirtIO Disk 1` --> `Disk bus` and changing the bus type to `SCSI`

* In the `KVM` `VM`, enable the following firewall ports for communication with the replication appliance:

```bash
sudo firewall-cmd --add-port=135/tcp --permanent
sudo firewall-cmd --add-port=443/tcp --permanent
sudo firewall-cmd --add-port=1024-65535/tcp --permanent
sudo firewall-cmd --add-port=443/udp --permanent
sudo firewall-cmd --reload
```

* Shut down the `VM`

* Remove the `SATA CDROM 1` hardware.

* Change the `SCSI Disk 1` `XML` to the following:

```xml
<disk type="file" device="disk">
  <driver name="qemu" type="qcow2"/>
  <source file="/PATH/TO/<YOUR_QCOW2_IMAGE>.qcow2"/>
  <target dev="sda" bus="scsi"/>
  <address type="drive" controller="0" bus="0" target="0" unit="0"/>
</disk>
```

* Power up the `VM` and verify it boots successfully.

### Step 2: prepare the Windows Server 2022 VM for the replication appliance

* Deploy a `Windows Server 2022` `VM` on `KVM` with 16GB of `RAM`, 8 `vCPUs` and 700GB of disk space. Choose the `Standard Desktop` installation.

* Set a password and then login to the Windows Server.

### Step 3: configure the Azure virtual network

* Go to `Virtual Networks` in the `Azure Portal` and click `Create`

* Under `Basics`, select your `Resource Group`, enter a `Virtual network name` and choose your region.

* For `Security`, leave all options at default and click `Next`

* For `IP addresses`, leave all options at default and click `Next`

* In `Tags`, apply your tags and click `Next`

* Click `Create`

### Step 4: set up the Azure Migrate project

* Go to `Azure Migrate` --> `Servers, databases and web apps` and select `Create project`

* Add your `Resource Group`, `Project Name`, `Geography` and click `Create`

* Under `Assessment tools`, go to `Migration and modernization` (**note, do not use the "Azure Migrate: Discovery and assessment" option
**) and click `Discover`

* For `Where do you want to migrate to?`, choose `Azure VM`

* For `Are your machines virtualized?`, select `Physical or other (AWS, GCP, Xen, etc.)`

* Select your `Target region`

* Check the box that says `Confirm that the target region for migration is "<REGION>"`

* Click `Create resources`

* Return to `Discover` and specify the same options for `Where do you want to migrate to?`, `Are your machines virtualized?` and `Target region`

* For `Experience type`, select `Simplified Experience (Recommended)`

* Go to Step 2 and click `Generate key` to create your `ASR replication appliance key`

* Store this key in a safe location for later use in the process.

### Step 5: Install and Configure the Replication Appliance

* Download the installer from the provided link to your `Windows Server 2022` `VM`

* Extract the zip file's contents.

* Open a `PowerShell` window as an Administrator, navigate to the folder and run `.\DRInstaller.ps1`

* If you receive an error about the script not being digitally signed, run this command instead and modify the path for your folder's location:

```text
powershell -ExecutionPolicy Bypass -File "C:\Users\Administrator\Downloads\DRAppliance\DRAppliance\DRInstaller.ps1"
```

* Go to the desktop and open the shortcut to the `Microsoft Azure Appliance Configuration Manager`

* Click `Continue` through the `Set up prerequisites`, `Appliance components` and `Select Replication appliance connectivity` steps.

* Under `Register with Recovery Services vault`, provide a friendly name and the `ASR replication appliance key` you generated earlier.

* Click `Login` and then `Copy code and Login`

* Log in with your `Azure` credentials.

* Once the appliance is registered, click `Continue`

* Under `Configure vCenter details`, check `I do not have vCenter Server/vSphere ESXi server. I'll protect my servers by manually discovering them using IP addresses.` and press `Continue`

* For `Provide physical server credentials`, click `Add credentials`

* Select the `Operating System` as `Linux`, provide a friendly name and input the `root` user's credentials.

* Click `Add`

* Under `Provide physical server information`, click `Add server`

* Input the `IP` address of your `Rocky Linux 8.10` `VM` and click `Add`

* Click `Continue` and wait approximately 30 minutes for the process to complete.

* Verify you see a green checkmark with `Completed appliance configuration successfully.`

### Step 6: configure replication

* In the `Azure Portal`, go to `Azure Migrate: Server Migration` and click `Replicate`

* Leave other options as default and for `Are your machines virtualized?` choose `Physical or other (AWS, GCP, Xen, etc.)`

* From `On-premises appliance`, choose your appliance and select `Continue`

* For `Basics` --> `Guest credentials`, choose your `Rocky Linux 8.10` `VM's` guest credentials.

* Select the checkbox next to your `VM` when it appears and click `Next`

* For `Target settings`, choose your `Virtual network` created earlier, leave other options as default and click `Next`

* At `Compute`, click `Next`

* For `Disks` --> `Disk Type`, select `Standard SSD` (or another disk type of your choice) and click `Next`

* For `Tags`, add your tags and click `Next`

* At `Review + Start replication`, choose `Replicate`

### Step 7: perform the migration

* Skip the `Test migration` step if not needed and go to `Migrate`

* Leave the default `Azure VM` option and click `Continue`

* Select the `VM` you want to migrate and leave `Yes, shutdown virtual machines(Ensures no data loss)` checked.

* Click `Migrate`

### Step 8: configure network access for the migrated VM

* Once complete, your local `VM` will shut down.

* Go to `Virtual Machines` in `Azure Portal` to access your newly migrated `VM`

* Navigate to your newly created `VM` under `Virtual Machines`, then `Networking` and then `Network settings`

* Click `Add network security group`

* When the `network security group` is created, click `Create port rule` and select `Inbound port rule`

* Specify an `IP` address, source and destination ports, the protocol to use, and the `Action` to allow or deny traffic.

* Click `Save`

* For temporary `SSH` access testing, you can set `Source` to `Any`, `Source port ranges` to `*`, `Destination port ranges` to `22`. Please check with your Security Team on how ports, protocols and IPs should be set up first for best security practices.

* Click `Add`

### Step 9: configure a public IP address

* Under `Networking` --> `Network settings`, check that a `Public IP address` is listed.

* If not present, click `(Configure)` and click on your `NIC` listed in the table. An example `NIC` name is `nic-rocky-linux-810-00-ipconfig`

* Check the box next to `Associate public IP address` and click `Save`

* Refresh the `IP configurations` page to see the `Public IP Address`

* Go to your `VM`, then `Connect` and select `Native SSH`

* `ssh` into your machine with the `root` account or another configured account.

## Notes

The migration process may take considerable time depending on the size of your source `VM` and network bandwidth.

Always perform a test migration before executing the final migration in production environments.

## References & related articles

[Set up the appliance through PowerShell](https://learn.microsoft.com/en-us/azure/site-recovery/deploy-vmware-azure-replication-appliance-modernized#set-up-the-appliance-through-powershell)

[Azure Site Recovery Appliance Installer](https://aka.ms/V2ARcmApplianceCreationPowershellZip)


Migrating a Rocky Linux KVM VM to Microsoft Azure


## Introduction

OpenLDAP clients and servers are capable of using the Transport Layer Security (TLS) framework to provide integrity and confidentiality protections and to support LDAP authentication using the SASL EXTERNAL mechanism. When using an unencrypted port for LDAP authentication, clients may be able to authenticate, however it may not be the right setting based on your security policies.

## Problem

When LDAP authentication occurs over an unencrypted port, certificate validation issues may be bypassed leading to a successful authentication. OpenLDAP servers, by default, demand certificate verification, which can cause failures if the certificate is self-signed or untrusted.

## Symptoms

LDAP authentication failures when using the encrypted port and not when using the unencrypted port may produce errors such as:

```text
7: AUTH: Bind failed ß------- This is where the connection breaks
7: AUTH: LDAP ERROR: -1: Can't contact LDAP server
```

## Resolution

To modify the client TLS settings, update the LDAP client configuration. Open the LDAP configuration file in a text editor:

```shell
sudo vi /etc/openldap/ldap.conf
```

Locate the `TLS_REQCERT` directive and modify its value based on your security requirements.

To bypass certificate validation completely (not recommended for secure environments):

```text
TLS_REQCERT never
```

To allow authentication with an untrusted certificate without disabling validation completely:

```text
TLS_REQCERT allow
```

Save the file and exit the editor. Restart any services dependent on LDAP authentication, or reboot the system if necessary.

## Root Cause

This issue arises because OpenLDAP servers, by default, demand certificate verification (`TLS_REQCERT demand`). If the certificate is self-signed or untrusted, authentication fails unless this setting is adjusted.

## Notes

The default OpenLDAP behavior (`TLS_REQCERT demand`) is intended for secure environments. Changing this setting can reduce security.

If modifying `TLS_REQCERT`, ensure other security measures are in place to prevent unauthorized access.

## References & Related Articles

[OpenLDAP TLS documentation](https://www.openldap.org/doc/admin26/tls.html)


OpenLDAP Fails to Authenticate Over Encrypted Port


## Introduction

CIQ had a customer issue, where the customer was attempting to set up local users and then authenticate using `Kerberos` to their `Active Directory` server. The following guide will explain how this was resolved.

## Problem

The customer was unable to authenticate via `ssh` to their Rocky Linux 9.2 LTS node using a local user account stored on the node. No passwords were locally saved in the `/etc/shadow` file and the passwords were only available on the `Active Directory` server.

The customer's configuration of `sssd`, `Kerberos` and `Active Directory` had worked without issue on CentOS 7.9, CentOS 8.1 and Rocky Linux 8.5. It was only on Rocky Linux 9.2 LTS where the customer was facing issues.

## Symptoms

When attempting to log in to the server via `ssh`, the following authentication messages were observed under `/var/log/secure`:

```bash
Dec 9 11:07:31 <server_name> sshd[1155]: Received signal 15; terminating.
Dec 9 11:07:31 <server_name> sshd[79752]: Server listening on 0.0.0.0 port 22.
Dec 9 11:07:31 <server_name> sshd[79752]: Server listening on :: port 22.
Dec 9 11:10:04 <server_name> unix_chkpwd[79934]: check pass; user unknown
Dec 9 11:10:04 <server_name> unix_chkpwd[79934]: password check failed for user (<username>)
Dec 9 11:10:04 <server_name> sshd[79919]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<ip_address> user=<username>
Dec 9 11:10:44 <server_name> sshd[79919]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<ip_address> user=<username>
Dec 9 11:10:44 <server_name> sshd[79919]: pam_sss(sshd:auth): received for user <username>: 4 (System error)
```

Account retrieval errors could also be seen via the `/var/log/sssd/sssd_pam.log` and `/var/log/sssd/sssd_nss.log` files:

```bash
[root@<server_name> ~]# cat /var/log/sssd/sssd_pam.log
(2024-12-16 13:45:58): [pam] [server_setup] (0x3f7c0): Starting with debug level = 0x0070
(2024-12-16 13:47:20): [pam] [cache_req_common_process_dp_reply] (0x3f7c0): [CID#1] CR #3: Could not get account info [<account_number>]: SSSD is offline

[root@<server_name> ~]# cat /var/log/sssd/sssd_nss.log
(2024-12-16 13:45:58): [nss] [server_setup] (0x3f7c0): Starting with debug level = 0x0070
(2024-12-16 13:47:20): [nss] [cache_req_common_process_dp_reply] (0x3f7c0): [CID#5] CR #17: Could not get account info [<account_number>]: SSSD is offline
```

## Resolution

The customer was utilizing `id_provider = files` in their `/etc/sssd/sssd.conf` file. Since the Rocky Linux 9.x series, [this is a deprecated option in sssd](https://sssd.io/docs/files-provider-deprecation.html).

Instead, they added the following settings into their `sssd.conf` file:

```bash
id_provider = proxy
krb5_server = example.com:88
proxy_lib_name = files
```

Then restarted the `sssd` service:

```bash
systemctl restart sssd`
```

The customer's `sssd.conf` file ultimately looked like the following (all sensitive customer information has been obfuscated):

```bash
[sssd]
    services = nss, pam
    domains = EXAMPLE.COM
    default_domain_suffix = example.com

[domain/EXAMPLE.COM]
    id_provider = proxy
    proxy_lib_name = files
    auth_provider = krb5
    krb5_realm = EXAMPLE.COM
    krb5_server = example.com:88
    krb5_validate = false
    krb5_ccachedir = /var/tmp
    krb5_keytab = /etc/krb5.keytab

[nss]
filter_groups = root,<local user>
filter_users = root,<local user>
```

In addition, the customer's `/etc/kr5b.conf` file:

```bash
includedir /etc/krb5.conf.d/

[logging]
    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

[libdefaults]
    default_realm=EXAMPLE.COM
    dns_lookup_realm = true
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    default_ccache_name = KEYRING:persistent:%{uid}

[realms]
     EXAMPLE.COM = {
     kdc = example.com:88
     admin_server = example.com:749
     }
     example.com = {
     kdc = example.com:88
     admin_server = example.com:749
     }

[domain_realm]
      example.com = EXAMPLE.COM
      example.com = EXAMPLE.COM

[appdefaults]
 pam = {
   debug = true
   ticket_lifetime = 36000
   renew_lifetime = 36000
   forwardable = true
   krb4_convert = false
 }
```

Furthermore, the customer's `/etc/nsswitch.conf` file:

```bash
# In order of likelihood of use to accelerate lookup.
shadow:     files
hosts:      files dns myhostname

aliases:    files
ethers:     files
gshadow:    files
# Allow initgroups to default to the setting for group.
# initgroups: files
networks:   files dns
protocols:  files
publickey:  files
rpc:        files
```

## Troubleshooting notes

Adding `debug_level = 9` to each section of the `sssd.conf` file, yields more verbose logs under `/var/log/sssd/*`

When troubleshooting the issue, the following logs are essential:

```bash
/var/log/sssd/*
/var/log/secure
```

In addition, these files are also important to check:

```bash
/etc/kr5b.conf
/etc/nsswitch.conf
/etc/pam.d/password-auth
/etc/pam.d/system-auth
/etc/sssd/sssd.conf
```

Next are these command outputs to run:

```bash
getent passwd <username>
klist
kinit <username>
ssh -o PubkeyAuthentication=no -vvv -k <username>@<server_name>
```

Finally, generating an `sosreport` from the node.

## References & related articles

[Mapping local users to Kerberos principals with SSSD](https://blog.oddbit.com/post/2015-07-16-mapping-local-users-to-kerbero/)  
[sssd.conf man page](https://linux.die.net/man/5/sssd.conf)  
[sssd-krb5 man page](https://linux.die.net/man/5/sssd-krb5)


Enabling Local User Login with Active Directory Authentication Using Kerberos


## Introduction

Optimizing NVIDIA GPU performance on Rocky Linux 8 is essential for computationally intensive tasks.

Performance inconsistencies can emerge, particularly when GPUs enter low-power states or incorrectly fall back to CPU rendering, significantly impacting workloads.

## Problem

NVIDIA GPUs have been known to intermittently drop performance to very low levels due to incorrect power states.

## Symptoms

Users experience initially acceptable GPU performance that rapidly deteriorates.

Tasks such as rotating graphical models take multiple seconds, indicating a fallback to CPU rendering or low GPU utilization.

Monitoring tools such as with `nvidia-smi` report sometimes observe GPUs stuck in performance mode P8 (low-power idle state) and show GPU utilization as minimal or zero during running of workloads.

## Resolution

### Environment Prerequisites:

* Rocky Linux 8

* NVIDIA GPU

* `root` or `sudo` privileges

* For NVIDIA GPU driver installation, [please follow this guide](https://docs.rockylinux.org/desktop/display/installing_nvidia_gpu_drivers/).

### Steps to Resolution:

The example steps below were tested using an NVIDIA 1060 GPU on Rocky Linux 8.10.

* Verify and set optimal tuned profile (for an HPC setup, the recommended profile is `hpc-compute`):

```bash
tuned-adm list
sudo tuned-adm profile hpc-compute
tuned-adm active
```

* Set GPU persistence and P-State permanently:

```bash
echo 'options nvidia NVreg_RegistryDwords="RMForcePstate=0"' | sudo tee /etc/modprobe.d/nvidia.conf
sudo dracut --force
sudo reboot
```

* GPU state checked with `nvidia-smi` before the above changes were made (observe that `P8` is listed, equivalent to `Basic HD video playback`):

```bash
|   0  NVIDIA GeForce GTX 1060 6GB    Off |
| 40%   35C    P8             10W /  120W |
```

* GPU state after the above changes with `nvidia-smi` (observe that the P-State has now changed to `P0` or `Maximum 3D performance`):

```bash
|   0  NVIDIA GeForce GTX 1060 6GB    Off |
| 40%   35C    P0             24W /  120W |
```

## Root cause

Due to incorrect GPU persistence settings, the GPU fails to be permanently set in a high power state.

## Notes

* Useful host-specific performance troubleshooting commands:

```bash
sudo dmidecode -t bios
sudo cpupower frequency-info
sudo lshw -class processor
sudo cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
tuned-adm active
tuned-adm list
```

* NVIDIA GPU-specific troubleshooting commands:

```bash
nvidia-smi
nvidia-smi -q -d PERFORMANCE
nvidia-smi -q -d POWER
nvidia-smi -q | grep -i "Persistence Mode"
nvidia-smi -q -d CLOCK
nvidia-smi --query-gpu=clocks.current.graphics,clocks.current.sm,clocks.current.memory --format=csv
```

* Ensuring persistent high-performance GPU states helps to resolve the majority of performance problems in computational and graphical workloads.

## References & related articles

[NVIDIA GPU Driver Installation on Rocky Linux](https://docs.rockylinux.org/desktop/display/installing_nvidia_gpu_drivers/)  
[NVIDIA GPU P-State Descriptions](https://docs.nvidia.com/gameworks/content/gameworkslibrary/coresdk/nvapi/group__gpupstate.html)


Maximizing NVIDIA GPU Performance on Rocky Linux 8


## Introduction

Frequently, businesses will want to migrate between major versions of their Linux distribution of choice in order to stay ahead of security threats, take advantage of improved hardware compatibility in a newer kernel, and other reasons for why you would want to move to a newer OS version.

This article will go over how to migrate configurations, system settings, users and more from an older system to a newer system.

**_⚠️ WARNING_** _The Rocky Linux project does not support in-place upgrades between major versions of Rocky Linux. This is also CIQ's stance and we recommend migrating between two or more nodes or in the case of a single node, backing up the node's contents and performing a fresh install._

**_⚠️ WARNING_** _Once you complete the migration, CIQ recommends performing a human review of the migrated configurations before applying them to the new system. The reason for this is that there are thousands of changes between major Rocky Linux versions, including removed and added packages, as well as version jumps of core system components such as `kernel` and `gcc`. Incompatible packages and system configurations will happen, which is why a manual review is required._

**_⚠️ WARNING_** _The playbooks linked here have been tested on two Azure Rocky Linux VMs, as well as two baremetal servers. The testing included a migration from Rocky Linux 8 to 9 and also from 8 to 10. The playbooks are not meant to be comprehensive, and CIQ recommends expanding and modifying the playbooks to suit your environment._

## Problem

You are moving between major Rocky Linux versions. You have two or more nodes. One node is running a newer version of Rocky Linux, while the other node is running an older version of Rocky Linux, which is either experiencing hardware failure or the OS reaching end of life. You want to migrate `users`, `systemd` services, `cron jobs`, and more to the newer system, which can be a tedious and drawn-out process when done manually.

## Resolution

### Prerequisites

* Two nodes - one running an older major version of Rocky Linux and the other running a newer version.

* A separate machine configured as the Controller, responsible for orchestrating the migration between the two Rocky Linux nodes using Ansible.

### Ansible setup

Install Ansible if you have not already done so on a separate machine that is not the Source Node or Target Node. This machine will be the Controller and will orchestrate the migration between the two nodes. We recommend using `pip` to install Ansible:

```bash
# Install python3 and pip if not already available:
sudo dnf install -y python3 python3-pip
 
# Install Ansible via pip for your user:
pip3 install --user ansible

# Install system-wide if required:
sudo pip3 install ansible
```

* Ensure the required Ansible Galaxy collections are installed:

```bash
ansible-galaxy collection install -r requirements.yml
```

### Migration playbook setup

* Place all configuration files [available here](https://sftp.ciq.com/web/client/pubshares/pjKMPp5SVXM7FuXBT6b3tT?compress=false) into the same directory.

* Please go through and edit any files to better match your environment. The files that contain `<>` characters require input. An example set of playbooks has been included in the above tarball to help you.

* A key file is the `inventory.ini` file with the Source Node and Target Node IP addresses and user (ideally `root` or a user with `sudo` privileges) that you want to login with.

### Migration testing

* Copy your SSH public key to both Nodes:

```bash
ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@<SOURCE_NODE_IP>
ssh-copy-id -i ~/.ssh/id_rsa.pub <USERNAME>@<TARGET_NODE_IP>
```

* Check that both nodes can be reached: 

```bash
ansible all -i inventory.ini -m ping
```

* You should see a similar result to the following output:

```bash
howard@ciq-rocky-linux10:~/ansible-baremetal-testing$ ansible all -i inventory.ini -m ping
source | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
target | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
```

* If the ping checks fail, ensure that `python3` is installed on all Nodes, that public and private ssh keys are available, and that the details under `inventory.ini` are correct. If going from an older version of Rocky Linux such as Rocky Linux 8.10, install a later version of `python` from the `powertools` repository:

```bash
sudo dnf config-manager --set-enabled powertools
sudo dnf install -y python39 
```

* The version of `python3` used has to be later than `python3.6`, which is installed by default on Rocky Linux 8.10 unless a newer version of `python` is installed.

* Run the `preflight-check.yaml` playbook and a total of `6` checks should report back as healthy regarding disk space availability, network connectivity, system information, and more :

```bash
ansible-playbook -i inventory.ini preflight-check.yml
```

### Migrating the data

* Start the migration between the Nodes:

```bash
ansible-playbook -i inventory.ini migrate-rocky.yml
```

* After the final phase of `Phase 5 - Cleanup SSH access` is finished, the migration will be complete.

* On the Target Node, a full report of the migration can be found under `/root/MIGRATION_COMPLETE_<UNIX_TIME>.txt`.

* Under the chosen user account's home directory of the Target Node (in this case `/root/migration_review`), you'll find migrated firewall, network, services and `ssh` configurations ported over. 

* For `/tmp/filtered_packages.txt` on the Target Node, this file has a list of the core system packages that came from the Source Node. Further exports such as security, users and system configurations can be found in `/tmp/system_export/`

### Human review

Please review the data that has been migrated over to the new node and work with your System Administrators to safely apply as many of the configurations as possible, without creating additional incompatibility errors.

## References & related articles

[Learning Ansible with Rocky Series - Rocky Linux Documentation](https://docs.rockylinux.org/books/learning_ansible/00-toc/)

[Rocky supported version upgrades - Rocky Linux Documentation](https://docs.rockylinux.org/guides/update_versions/)


Migrating Between Major Rocky Linux Versions Using Multiple Nodes


## Introduction

If you are on an LTS version of Rocky Linux and would like to transition to the latest community version of Rocky Linux, you need to revert the repositories and `release` back to defaults.

## Resolution

For example, if you were on the Rocky Linux 9.2 LTS and wanted to upgrade to the community Rocky Linux 9.6, you can accomplish this by running these commands:

```
sudo depot disable lts-9.2
sudo dnf swap ciq-lts92-rocky-release rocky-release --allowerasing --skip-broken
sudo dnf upgrade
```

## Notes

- The `dnf swap` command will replace all of your Rocky Linux repositories in `/etc/yum/repos.d/` with the default repositories. If you use any custom repository URLs, make sure to back them up before running this command.
- The `dnf upgrade` command will upgrade your node to Rocky Linux version 9.6. Please ensure you have the necessary backups before proceeding.
- If you use `secure boot`, test this on a non-production server first, to ensure there are no issues before upgrading.

## References & related articles

[CIQ Long Term Support](https://ciq.com/services/long-term-support/)


Migrate From Rocky Linux LTS to the Latest Rocky Linux Community Version


## Introduction

MPI (Message Passing Interface) benchmarking is heavily utilized in the automobile industry and is a backbone of High Performance Computing. With benchmarking, the objective is to pass a message between two nodes as quickly as possible. The faster the message is passed between both nodes, the more parallel operations you will be able to perform on your selected hardware.

This article will cover how to set up MPI benchmarking on a Rocky Linux 8.10 image on two nodes in Azure.

## Prerequisites

* Two individual VMs or a Virtual Machine Scale Set (Virtual Machine Scale Set) (a VMSS is the recommended option and is what is used in this article). 

* The `size` of the VMs / VMSS needs to be able to support Infiniband and RDMA. Example `sizes` that support Infiniband and RDMA are `HBv4`, `H16r` and so on (the testing performed for this article was on a Virtual Machine Scale Set using two `HB176-24rs_v4` instances). You may need to increase your quota in your Azure subscription in order to access `sizes` such as the `HBv4`. 

* Utilize a Rocky Linux VHD to be applied to each node (for this documentation, this was tested with Rocky Linux 8.10). The image requires the `cld_next` kernel, as well as the `waagent`, `OFED`, Infiniband, and RDMA drivers. In addition, `Open MPI` needs to be installed for the MPI benchmarking to be performed.

## Azure setup

All steps are performed using the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?view=azure-cli-latest&pivots=apt).

### Step 1: create a resource group

```bash
az group create --name <RESOURCE-GROUP_NAME> --location <LOCATION> 
```

### Step 2: generate a storage account

```bash
az storage account create \
    --name <NAME> \
    --resource-group <RESOURCE-GROUP_NAME> \
    --location <LOCATION> \
    --sku Standard_LRS
```

### Step 3: provision a container

```bash
az storage container create \
    --account-name <NAME> \
    --name <NAME>
```

### Step 4: retrieve the storage account key and assign it to the `STORAGE_KEY` variable

```bash
STORAGE_KEY=$(az storage account keys list \
    --resource-group <RESOURCE-GROUP_NAME> \
    --account-name <NAME> \
    --query '[0].value' -o tsv)
```

### Step 5: display the key for verification (optional)

```bash
echo $STORAGE_KEY
```

### Step 6: upload the Rocky Linux VHD image from your local filesystem

```bash
az storage blob upload \
    --account-name <NAME> \
    --account-key $STORAGE_KEY \
    --container-name <NAME> \
    --type page \
    --file <VHD_IMAGE> \
    --name <VHD_IMAGE>
```

### Step 7: set the VHD URI to the `VHD_URI` variable

```bash
VHD_URI="https://<NAME>.blob.core.windows.net/<NAME>/<VHD_IMAGE>"
```

### Step 8: create a managed image

```bash
az image create \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <IMAGE_NAME> \
    --os-type Linux \
    --source $VHD_URI \
    --location <LOCATION>
```

### Step 9: generate an ssh key pair for accessing the nodes

```bash
# Generate an ssh key pair if you don't have one already on your local machine
ssh-keygen -t rsa -b 4096 -f ~/.ssh/<SSH_KEY> -N ""

# Check the public key for verification
cat ~/.ssh/<SSH_KEY>.pub
```

### Step 10: create a proximity placement group

```bash
az ppg create \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <PROXIMITY_PLACEMENT_GROUP_NAME> \
    --location <LOCATION> \
    --type Standard
```

### Step 11: Create a virtual machine scale set 

```bash
az vmss create \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NAME> \
    --image <IMAGE_NAME> \
    --instance-count 2 \
    --vm-sku <SIZE> \
    --admin-username azureuser \
    --ssh-key-values ~/.ssh/<SSH_KEY>.pub \
    --upgrade-policy-mode automatic \
    --lb <LOAD_BALANCER_NAME> \
    --backend-pool-name <BACKEND_POOL_NAME> \
    --location <LOCATION> \
    --public-ip-per-vm \
    --orchestration-mode Uniform \
    --ppg <PROXIMITY_PLACEMENT_GROUP_NAME>
```

### Step 12: assign public ip addresses for each node

```bash
az network public-ip create \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NODE_1_PUBLIC_IP_NAME> \
    --allocation-method Static \
    --sku Standard \
    --location <LOCATION>

az network public-ip create \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NODE_2_PUBLIC_IP_NAME> \
    --allocation-method Static \
    --sku Standard \
    --location <LOCATION>
```

### Step 13: remove any existing nat rules

```bash
az network lb inbound-nat-rule delete \
    --resource-group <RESOURCE-GROUP_NAME> \
    --lb-name <LOAD_BALANCER_NAME> \
    --name NatRule
```

### Step 14: create a nat pool

```bash
az network lb inbound-nat-pool create \
    --resource-group <RESOURCE-GROUP_NAME> \
    --lb-name <LOAD_BALANCER_NAME> \
    --name <NAT_POOL_NAME> \
    --protocol Tcp \
    --frontend-port-range-start 50000 \
    --frontend-port-range-end 50119 \
    --backend-port 22 \
    --frontend-ip-name <LOAD_BALANCER_FRONTEND_NAME>
```

### Step 15: add the nat pool to the vmss:

```bash
az vmss update \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NAME> \
    --add virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerInboundNatPools \
    '{
      "id": "/subscriptions/2247734c-d128-46cd-a462-ee14c4302d9b/resourceGroups/<RESOURCE-GROUP_NAME>/providers/Microsoft.Network/loadBalancers/<LOAD_BALANCER_NAME>/inboundNatPools/<NAT_POOL_NAME>"
    }'
```

### Step 16: check the vmss configuration:

```bash
az vmss show \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NAME> \
    --query "virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerInboundNatPools" \
    --output json
```

### Step 17: check that the nat rules were created:

```bash
az network lb inbound-nat-rule list \
    --resource-group <RESOURCE-GROUP_NAME> \
    --lb-name <LOAD_BALANCER_NAME> \
    --output table
```

### Step 18: verify each node's nics:

```bash
az vmss nic list \
    --resource-group <RESOURCE-GROUP_NAME> \
    --vmss-name <NAME> \
    --query "[].{Name:name, PrivateIP:ipConfigurations[0].privateIpAddress}" \
    --output table
```

### Step 19: get detailed information on the nics' ip configuration:

```bash
az vmss nic list \
    --resource-group <RESOURCE-GROUP_NAME> \
    --vmss-name <NAME> \
    --query "[].ipConfigurations[0]" \
    --output json
```

### Step 20: get the vmss instance information

```bash
az vmss list-instances \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NAME> \
    --output table
```

### Step 21: display the public ip of each node

```bash
az vmss list-instance-public-ips \
    --resource-group <RESOURCE-GROUP_NAME> \
    --name <NAME> \
    --output table
```

### Step 22: ssh into each node

```
ssh -i ~/.ssh/<SSH_KEY> azureuser@<NODE_IP>
```

## For each node, perform the below steps:

### Step 23: generate an ssh key pair

```bash
ssh-keygen -t rsa -b 4096 -f ~/.ssh/inter_instance_key -N ""
```

### Step 24: share the public keys between both nodes

```bash
# On node 1, show the public key
cat ~/.ssh/inter_instance_key.pub

# Copy the public key

# SSH into node 2 and add node 1's public key to node 2's authorized_keys file:
echo "PASTE_NODE_1_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys

# On node 2, display the public key
cat ~/.ssh/inter_instance_key.pub

# Copy the public key

# SSH into node 1

# Add node 2's public key to node 1's authorized_keys file:
echo "PASTE_NODE_2_PUBLIC_KEY_HERE" >> ~/.ssh/authorized_keys
```

### Step 25: create the hostlist.txt file for both nodes

```bash
# Get hostnames of both nodes

# On each node, create the hostlist.txt file

# First, get the hostnames of each node (run the below command on each node to identify the hostname)
hostname

# Create hostlist.txt with both hostnames
cat > ~/hostlist.txt << EOF
<NODE_1_HOSTNAME>
<NODE_2_HOSTNAME>
EOF
```

### Step 26: generate the ssh config file on both nodes

```bash
# Create ~/.ssh/config file on each instance
cat > ~/.ssh/config << 'EOF'
Host <START_OF_HOSTNAME_PATTERN>*
  IdentityFile /home/azureuser/.ssh/inter_instance_key
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
EOF
```

### Step 27: set the 600 permissions for the ssh config file on both nodes, to ensure that only the owner has read / write permissions

```bash
chmod 600 ~/.ssh/config
```

### Step 28: set 600 permissions for private ssh key

```bash
chmod 600 ~/.ssh/inter_instance_key
```

### Step 29: test ssh connectivity between the nodes

```bash
# From node 1 to node 2
ssh -i ~/.ssh/inter_instance_key <NODE_2_HOSTNAME>

# From node 2 to node 1
ssh -i ~/.ssh/inter_instance_key <NODE_1_HOSTNAME>
```

### Step 30: rdma setup for the waagent on both nodes 

```bash
sudo vi /etc/waagent.conf

# Ensure the following are enabled and uncommented
OS.EnableRDMA=y
OS.CheckRdmaDriver=y

# Save and verify the changes
grep -i rdma /etc/waagent.conf | grep -v "^#"
```

### Step 31: restart the waagent service

```bash
sudo systemctl restart waagent
sudo systemctl daemon-reload
```

### Step 32: load the ib_ipoib module

```bash
sudo modprobe ib_ipoib
```

### Step 33: configure the infiniband network

```bash
# On node 1, set an ip address (in the below example, it is 172.20.0.10)
sudo ip addr add 172.20.0.10/24 dev ib0

# Bring the link up
sudo ip link set ib0 up

# Verify the infiniband interface is up
ibdev2netdev

# On node 2, set an ip address
sudo ip addr add 172.20.0.11/24 dev ib0

# Bring the link up
sudo ip link set ib0 up

# Verify the interface is up
ibdev2netdev
```

### Step 34: test infiniband connectivity

```bash
# From node 1 to node 2 with the above example ips
ping -c 3 172.20.0.11

# From node 2 to node 1
ping -c 3 172.20.0.10
```

### Step 35: run the mpi benchmarking tests

Confirm that messages can be passed between the two nodes successfully with the `pingpong` and `allreduce` benchmarks. 

```bash
# pingpong benchmark
/usr/mpi/gcc/openmpi-4.1.7rc1/bin/mpirun -n 2 -N 1 -hostfile ~/hostlist.txt -x UCX_NET_DEVICES=mlx5_0:1 /usr/mpi/gcc/openmpi-4.1.7rc1/tests/imb/IMB-MPI1 -msglog 27:28 pingpong

# allreduce benchmark
/usr/mpi/gcc/openmpi-4.1.7rc1/bin/mpirun -n 72 -N 36 -hostfile ~/hostlist.txt -x UCX_NET_DEVICES=mlx5_0:1 /usr/mpi/gcc/openmpi-4.1.7rc1/tests/imb/IMB-MPI1 -npmin 72 allreduce
```

## Notes

### If ssh between the nodes fails

1. Verify hostnames are correct in the `hostlist.txt` file.

2. Check that public keys are properly added to `authorized_keys` file.

3. Ensure firewall rules allow internal communication.

4. Verify the ssh server service is running: `sudo systemctl status sshd`

### If Infiniband connectivity fails

1. Check if the `ib0` interface is visible: `ip addr show ib0`

2. Verify RDMA modules are loaded: `lsmod | grep rdma`

3. Check the Infiniband status: `ibstat`

4. Verify connectivity with `ibping`

### If the MPI tests fail

1. Ensure all nodes are reachable via SSH without password prompts.

2. Verify OpenMPI installation: `which mpirun`

3. Check that `UCX_NET_DEVICES` matches your hardware: `ucx_info -d`

## How to quickly cleanup a test setup and delete all resources

```bash
# Delete the entire resource group and all its resources
az group delete --name <RESOURCE-GROUP_NAME> --yes --no-wait

# Check the deletion status
az group show --name <RESOURCE-GROUP_NAME>
```

## References & related articles

[Open MPI](https://www.open-mpi.org/)


How to Perform MPI Benchmarking on Azure with Rocky Linux


## Introduction

When configuring MTU values on bonded interfaces in Rocky Linux using NetworkManager’s `nmcli`, you may find that the MTU changes aren't reflected when you inspect the interfaces (for example, via `ip a | grep mtu`). Although NetworkManager’s logs indicate that NetworkManager applies the MTU, the setting resets to the default value. This can happen when you incorrectly apply the MTU settings to the interface instead of the bond.

## Problem

The issue occurs because the MTU is only set on the slave interfaces and not on the bond interface itself. NetworkManager propagates MTU settings from the bond interface down to its slaves and any VLANs on top of it, but it doesn't propagate in the opposite direction. Without an MTU on the primary bond connection, slave interfaces inherit the default MTU.

## Symptoms

You may observe all or some of the following:

- MTU values applied via `nmcli` don't appear in the output of `ip a | grep mtu`.

- NetworkManager journal logs report the MTU change but the interfaces retain the default MTU.

- Slave interfaces such as eth1 and eth2 show the default MTU instead of the desired value.

- VLAN interfaces on top of the bond don't inherit the configured MTU.

## Resolution

### Correct configuration approach

The key to resolving this issue is to ensure the MTU is properly set in the bond interface configuration file, as MTU settings propagate from the bond interface to its slaves, but not in reverse.

#### Edit the bond interface configuration

```bash
sudo vi /etc/NetworkManager/system-connections/bond0.nmconnection
```

#### Add the MTU setting under the `[ethernet]` section

```ini
[ethernet]
mtu=9000
```

#### Remove any MTU settings from the slave interface configuration files

```bash
sudo vi /etc/NetworkManager/system-connections/eth1.nmconnection
sudo vi /etc/NetworkManager/system-connections/eth2.nmconnection
# Remove any `mtu=` lines from these files
```

### Applying the changes

There are several methods to apply these changes:

#### Reload the configuration without disruption

```bash
sudo nmcli con load /etc/NetworkManager/system-connections/bond0.nmconnection
sudo nmcli dev reapply bond0
```

This method attempts to update the device with changes to the active connection without disconnecting.

#### Restart the connection

```bash
sudo nmcli con down bond0
sudo nmcli con up bond0
```

This method is more disruptive but may be necessary if `nmcli dev reapply` doesn't work.

#### Restart NetworkManager

```bash
sudo systemctl restart NetworkManager
```

It's generally advisable to use more targeted approaches, such as reapplying the connection or bringing the interface down and back up, before opting for a full service restart.

### Verification

After applying the changes, verify the MTU settings are correctly applied:

```bash
ip a | grep mtu
```

The output should show the bond interface and all related interfaces (slaves and VLANs) with the specified MTU value.

## Notes

- MTU settings propagate from bond interfaces to their slaves, not the other way around.

- Restarting NetworkManager isn't intended to perform the same "down" and "up" actions as restarting network initscripts.

- For persistent changes, always make sure the configurations are properly set in the connection files.

- The `nmcli dev reapply` command attempts to update a device with changes made to the active connection since it was last applied.

## References

[Rocky Linux NetworkManager Gem](https://docs.rockylinux.org/10/gemstones/network/network_manager/)  
[Rocky Linux NetworkManager Documentation](https://docs.rockylinux.org/guides/network/basic_network_configuration/)


Configuring MTU Settings for Bonded Interfaces in NetworkManager


## Introduction

NFSv2 was completely removed in Rocky Linux 9. You must use Rocky Linux 8 if you desire NFSv2 support.

## Problem

NFSv2 cannot be mounted or configured on Rocky Linux 9 because support for this version of NFS has been completely removed from both the kernel and the userspace.

## Resolution

If you require NFSv2 support, the recommended solution is to migrate to Rocky Linux 8.10. This version includes support for NFS versions 2, 3, 4.0, 4.1, and 4.2. Rocky Linux 8.10 is fully supported and will remain so until May 31, 2029. CIQ offers further extended support for Rocky Linux 8.10 through its [Long-Term Support (LTS) program](https://ciq.com/services/long-term-support/).

While you can compile your own kernel for Rocky Linux 9 that supports NFSv2, CIQ doesn't recommend this.

## Notes

NFSv2 was removed because it’s an outdated, insecure protocol limited to 32-bit file sizes, lacking modern features like ACLs, and strong authentication. Its continued support added unnecessary complexity and security risk, so both user-space tools and kernel support were dropped in favor of NFSv3 and v4. Users are encouraged to migrate to newer NFS versions.

## References

[Rocky Linux NFS Guide](https://docs.rockylinux.org/labs/networking/lab5-nfs/?h=nfs)  
[Rocky Linux Support Timelines](https://wiki.rockylinux.org/rocky/version/)  
[Rocky Linux LTS from CIQ](https://ciq.com/services/long-term-support/)


NFSv2 Support Removed in Rocky Linux 9


## Introduction

NFSv3 file locking requires additional configuration compared to NFSv4. This article covers the proper setup of NFSv3 file locking on Rocky Linux.

## Problem

NFSv3 file locking operations are failing or timing out. Applications report issues obtaining file locks, and `flock` commands timeout on NFSv3 mounts.

## Resolution

### Verify rpc-statd service status

NFSv3 requires the `rpc-statd` service to be running on both the server and all clients for file locking to work properly.

Check the status of rpc-statd:

```bash
systemctl status rpc-statd
```

If the service isn't running, enable, and start it:

```bash
systemctl enable rpc-statd
systemctl start rpc-statd
```

### Configure fixed ports for NFS services

By default, NFSv3 RPC services use random ports, which can cause firewall issues. To configure fixed port numbers, edit the `/etc/nfs.conf` file and add or modify the following sections:

```ini
[lockd]
port=5555

[statd]
port=6666
```

### Configure firewall rules

If `firewalld` is enabled, whitelist the NFS services and the fixed ports:

```bash
firewall-cmd --permanent --add-service=rpc-bind
firewall-cmd --permanent --add-service=nfs
firewall-cmd --permanent --add-port={5555/tcp,5555/udp,6666/tcp,6666/udp}
firewall-cmd --reload
```

### Restart services

Restart the necessary services to apply the configuration changes:

```bash
systemctl restart rpc-statd nfs-server
```

### Verify configuration

To test file locking is working correctly, perform a simple test showing that one client waits for another to release the lock.

#### Test from Client 1

```bash
flock -x /path/to/nfs/mount/file.txt -c 'echo "Client 1 holding the lock"; sleep 30'
```

#### Test from Client 2 (run this while Client 1 is still holding the lock)

```bash
flock -x /path/to/nfs/mount/file.txt -c 'echo "Client 2 holding the lock"; sleep 10'
```

#### Expected behavior

**Client 1** should immediately display "Client 1 holding the lock" and hold the lock for 30 seconds.
**Client 2** should wait until Client 1's 30-second timer finishes before displaying "Client 2 holding the lock".

#### If file locking is working correctly

Client 2 should wait approximately 30 seconds before executing.

#### If file locking isn't working

Client 2 executes immediately without waiting for Client 1 to release the lock. Both clients may display their messages simultaneously.

## Root cause

NFSv3 uses the Network Lock Manager (NLM) sidecar protocol for file locking, which requires the `rpc-statd` service. Unlike NFSv4, which has file locking built into the protocol, NFSv3 relies on separate RPC services that must be properly configured and accessible through firewall rules.

## Notes

- If using a different NFS server implementation (like Dell OneFS), verify that the server also supports and has NLM properly configured.

- The `flock` testing only tests accurately between 2 NFS clients and not necessarily between a client and the server itself.

- To avoid potential data corruption, always access NFSv3 shares via NFS rather than locally on the server, since local and remote file locks don't interact.

## References & related articles

[Rocky Linux NFS Documentation](https://docs.rockylinux.org/guides/file_sharing/nfsserver/)  
[Using flock](https://linuxhandbook.com/flock-command/)


Configuring NFSv3 File Locking on Rocky Linux


## Introduction

The Nagios Remote Plugin Executor (NRPE) plugin is fundamental when combined with Nagios for adequate monitoring and provides you with system metrics including CPU, memory, disk usage and more from your Linux and Windows fleet.

This article will detail how to setup a basic Nagios Core server-client environment on Rocky Linux 9.5, how to install the NRPE plugin, and provide example NRPE commands with which to check the status of your Nagios clients.

## Prerequisites

* Two baremetal or virtual nodes running Rocky Linux 9.5, with at a minimal of 1 vCPU, 512MB RAM, and 500MB of disk space. The recommendation for Rocky Linux is 4 vCPU cores, 4096MB RAM and 10GB of disk space.

* Access to either the `root` account or a user with `sudo` privileges.

## Nagios Core server configuration

### Initial setup

* To install the latest version of Nagios Core Server, it is recommended to build directly from source, as opposed to downloading the packages from the EPEL repository.

* Upgrade all packages:

```bash
dnf upgrade -y
```

* Reboot the server:

```bash
reboot
```

* Install the required packages for Nagios:

```bash
dnf install -y gcc gd gd-devel glibc glibc-common httpd perl php openssl-devel wget
```

* Install the `Development Tools` group to aid with the source compilation process:

```bash
dnf groupinstall -y "Development Tools"
```

* Switch to the `tmp` directory:

```bash
cd /tmp
```

* Pull down the latest Nagios Core source code:

```bash
wget --output-document="nagioscore.tar.gz" $(wget -q -O - https://api.github.com/repos/NagiosEnterprises/nagioscore/releases/latest  | grep '"browser_download_url":' | grep -o 'https://[^"]*')
```

* Extract the tarball:

```bash
tar xzf nagioscore.tar.gz
```

* Change into the `nagios` directory (in this example, the version is `4.5.9`):

```bash
cd /tmp/nagios-4.5.9/
```

* Run `configure` to make sure all required dependencies are available:

```bash
./configure
```

* Run `make all` to compile the source code:

```bash
make all
```

* Generate the `nagios` user and group:

```bash
make install-groups-users
```

* Add the `apache` user that was generated from the previous `dnf` package installs into the `nagios` group:

```bash
usermod -a -G nagios apache
```

* Install the required binaries, CGIs and HTML files:

```bash
make install
```

* Install the `nagios` daemon:

```bash
make install-daemoninit
```

* Download the latest plugins pack:

```bash
wget --output-document="nagios-plugins.tar.gz" $(wget -q -O - https://api.github.com/repos/nagios-plugins/nagios-plugins/releases/latest  | grep '"browser_download_url":' | grep -o 'https://[^"]*')
```

* Unpack the tarball:

```bash
tar -xf nagios-plugins.tar.gz
```

* Change into the plugins directory, in this case it is version `2.4.12`:

```bash
cd nagios-plugins-2.4.12/
```

* Check for dependencies:

```bash
./configure
```

* Compile from source:

```bash
make
```

* Install the plugins to the appropriate directories:

```bash
make install
```

* Confirm the plugins were installed correctly. The number of plugins should be between `60~70`:

```bash
ls -l /usr/local/nagios/libexec/ | wc -l
```

* Set the `httpd` `systemd` service to restart on the next reboot:

```bash
systemctl enable httpd
```

* Similarly, configure the `nagios` service to start on a reboot:

```bash
systemctl enable nagios
```

* Setup `Command Mode`:

```bash
make install-commandmode
```

* Install the required configuration files:

```bash
make install-config
```

* Install the Apache web server files:

```bash
make install-webconf
```

* Configure `firewalld` to allow traffic on TCP port `80`:

```bash
firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --reload
```

* Create the `nagiosadmin` account and set a password:

```bash
htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin
```

* Start the `httpd` service:

```bash
systemctl start httpd
```

* Start the `nagios` service:

```bash
systemctl start nagios
```

### NRPE plugin installation

* Check the [NRPE plugin releases](https://github.com/NagiosEnterprises/nrpe/releases/) and download the latest version. As of writing, this is `4.1.3`:

```bash
wget https://github.com/NagiosEnterprises/nrpe/releases/download/nrpe-4.1.3/nrpe-4.1.3.tar.gz
```

* Extract the tarball:

```bash
tar -xf nrpe-4.1.3.tar.gz
```

* Change into the `nagios` directory (in this case version `4.1.3`):

```bash
cd nrpe-4.1.3/
```

* Check for dependency issues:

```bash
./configure
```

* Compile from source:

```bash
make all
```

* Install the plugin daemon and sample config files:

```bash
make install-daemon
make install-init
make install-plugin
```

## Nagios Core client configuration

### Initial setup

* Make sure all packages are up to date:

```bash
dnf upgrade -y
```

* Reboot the client machine:

```bash
reboot
```

* Install required packages:

```bash
dnf install -y automake autoconf gcc gd gd-devel gettext glibc glibc-common make net-snmp net-snmp-utils openssl-devel tar wget
```

* Generate the `nagios` user and assign a password:

```bash
useradd nagios
passwd nagios
```

* Change into the `tmp` directory:

```bash
cd /tmp
```

* Download the plugins pack:

```bash
wget --output-document="nagios-plugins.tar.gz" $(wget -q -O - https://api.github.com/repos/nagios-plugins/nagios-plugins/releases/latest  | grep '"browser_download_url":' | grep -o 'https://[^"]*')
```

* Extract the tarball:

```bash
tar -xf nagios-plugins.tar.gz
```

* Change into the plugins directory (in this case the version is `2.4.12`):

```bash
cd nagios-plugins-2.4.12/
```

* Check for dependency issues:

```bash
./configure
```

* Compile from source:

```bash
make
```

* Install the required packages:

```bash
make install
```

* Ensure the plugins were installed correctly (there are usually `60~70` plugins available):

```bash
ls -l /usr/local/nagios/libexec/ | wc -l
```

* Configure directory permissions for the `nagios` user:

```bash
chown nagios.nagios /usr/local/nagios
chown -R nagios.nagios /usr/local/nagios/libexec
```

### NRPE installation

* Pull down the latest version of the NRPE plugin (as mentioned above, the current version is `4.1.3`):

```bash
wget https://github.com/NagiosEnterprises/nrpe/releases/download/nrpe-4.1.3/nrpe-4.1.3.tar.gz
```

* Extract the tarball:

```bash
tar -xf ./nrpe-4.1.3.tar.gz
```

* Change into the `nagios` directory:

```bash
cd nrpe-4.1.3/
```

* Check for any dependency issues:

```bash
./configure
```

* Compile the NRPE daemon and client:

```bash
make all
```

* Install the plugin daemon and sample config files:

```bash
make install-config
make install-daemon
make install-plugin
```

* Create a `systemd` service for the NRPE daemon:

```bash
make install-init
```

* Enable the NRPE daemon `systemd` service to start on next boot:

```bash
systemctl enable nrpe
```

* Configure the NRPE plugin configuration file at `/usr/local/nagios/etc/nrpe.cfg` and add the address of your Nagios server. In the example illustrated below, the Nagios server address is `192.168.1.11`:

```bash
sed -i 's/allowed_hosts=127.0.0.1,::1/allowed_hosts=127.0.0.1,192.168.1.11/' /usr/local/nagios/etc/nrpe.cfg
```

* Open TCP port `5666` in `firewalld`:

```bash
firewall-cmd --zone=public --add-port=5666/tcp --permanent
firewall-cmd --reload
```

* Start the `nrpe` service:

```bash
systemctl start nrpe
```

* Verify that NRPE was installed successfully:

```bash
/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1
```

## Commands to check metrics on a Nagios client from the server

* In the following examples, the Nagios client has an IP of `192.168.1.12`:

* Initial check that the Nagios client can be reached:

```bash
/usr/local/nagios/libexec/check_nrpe -H 192.168.1.12
NRPE v4.1.3
```

* Check the amount of logged-in users:

```bash
/usr/local/nagios/libexec/check_nrpe -H 192.168.1.12 -c check_users
USERS OK - 1 users currently logged in |users=1;5;10;0
```

* Check the load on the client system:

```bash
/usr/local/nagios/libexec/check_nrpe -H 192.168.1.12 -c check_load
OK - load average per CPU: 0.00, 0.01, 0.01|load1=0.000;0.150;0.300;0; load5=0.007;0.100;0.250;0; load15=0.010;0.050;0.200;0;
```

## Notes

* The above configurations for server and client work on both FIPS and non-FIPS systems.

* `nagios` logs are located under `/usr/local/nagios/var/nagios.log`.

* Further logs and outputs can be found by running `journalctl -exu nrpe` and `netstat -at | grep nrpe`.

* Make sure the latest plugins pack is installed. If not, you will observe `NRPE: Unable to read output` in your terminal, when running commands such as `/usr/local/nagios/libexec/check_nrpe -H 127.0.0.1 -c check_load`.

## References & related articles (Optional)

* [Nagios Installation Guide for Rocky Linux](https://support.nagios.com/kb/article/nagios-core-installing-nagios-core-from-source-96.html#RHEL)

* [NRPE Plugin](https://exchange.nagios.org/directory/Addons/Monitoring-Agents/NRPE--2D-Nagios-Remote-Plugin-Executor/details)

* [NRPE Releases](https://github.com/NagiosEnterprises/nrpe/releases)


Installing the NRPE Plugin for Nagios Core on Rocky Linux


## Introduction

Customers migrating to Rocky Linux or using CIQ LTS support need reliable and accurate access to the correct software repositories.

This document provides the official and complete list of public mirrors and repositories that CIQ and Rocky Linux users should use.

It ensures customers can correctly configure their systems for `migrate2rocky`, CIQ LTS support, and package management across all supported versions.

## Problem

When customers migrate from Red Hat Enterprise Linux, CentOS, or other distributions, they may not know which mirror or repository to use. They risk accessing outdated or unsupported packages, or broken links during migration.

CIQ LTS customers specifically need to access their distribution's matching LTS repository version to ensure stability.

## Symptoms

Systems fail to update correctly after using the [migrate2rocky](https://docs.rockylinux.org/guides/migrate2rocky/) script.

`dnf` or `yum` errors referencing missing packages, 404s, or unexpected versions.

CIQ LTS customers pulling down incorrect Rocky Linux versions not aligned with their support agreement.

## Resolution

### Main Rocky Linux repositories

[download.rockylinux.org](download.rockylinux.org) and [dl.rockylinux.org](dl.rockylinux.org) are the official repositories for Rocky Linux.

### Rocky Linux public mirrors

For official mirrors, there is a list at [https://mirrors.rockylinux.org](https://mirrors.rockylinux.org)

Choose the Rocky Linux version and architecture you desire and then select the mirror that is geographically closest to you.

### CIQ LTS Customers

For CIQ LTS Customers, the format of the URL is `https://depot.ciq.com/files/<repository_name>/<repository_name>-<architecture>`

An example for the Rocky Linux 9.6 LTS `AppStream` would be `https://depot.ciq.com/files/lts-9.6/rocky-appstream-9.6.x86_64`

[Once CIQ Depot is installed](https://portal.ciq.com/docs/getting-started) and you have logged in, running `sudo depot list` will show the available repositories you can subscribe to.

## Root cause

Customers can unintentionally utilise mirrors that are either not in their geographical region, are not updated enough.  

This results in mismatched versions, unsupported configurations or broken links when pulling packages.

## Notes

Ensure to enable the correct firewall ports to allow access to the mirror of your choice. The ports will be TCP ports `80` for HTTP and `443` for HTTPS.

## References & related articles

[CIQ Depot Documentation](https://portal.ciq.com/docs/depot-guide)

[dl.rockylinux.org](dl.rockylinux.org)

[download.rockylinux.org](download.rockylinux.org)

[Integrating CIQ Repositories with Red Hat Satellite Guide](https://portal.ciq.com/docs/satellite-guide)

[Mirroring CIQ Repositories Using reposync Guide](https://portal.ciq.com/docs/reposync-guide)

[Mirroring CIQ Repositories Using rsync Guide](https://portal.ciq.com/docs/rsync-guide)

[Rocky Linux Mirror List](https://mirrors.rockylinux.org)

[Rocky Linux Repositories](https://wiki.rockylinux.org/rocky/repo/#notes-on-difference-between-x-and-xy-in-mirrors)


Accessing Mirrors for Rocky Linux and CIQ LTS Customers


## Introduction

A `mutex` (meaning `mutual exclusion`) prevents multiple threads from accessing the same resource simultaneously. When CIQ was running a `futex` test (a form of testing that checks the Linux kernel `futex` system call API), we observed `mutex` slowdown when running the `futex` testing on Rocky Linux 8.9 compared with CentOS 7.9.

We investigated this performance discrepancy further and this article will reveal why there was a performance differential between the two operating systems.

## Resolution

The performance differences between both operating systems were due to the BIOS settings on the HPE Blade machines that were used for testing. The CPU governor settings were set to `balanced power saving` mode.

Upon configuring the CPU governor to `static high performance` mode in the BIOS, we saw a dramatic increase in performance for Rocky Linux 8.9.

Running the `futex` tests again on both Rocky Linux 8.9 and CentOS 7.9 produced the performance results as expected.

We also advise setting the `tuned-adm` profile to `hpc-compute` to ensure maximum performance from Rocky Linux.

## Conclusion

When comparing performance between a Rocky Linux X.X version and another operating system, CIQ advises checking that the BIOS settings have the CPU governor configured for maximum performance.

In addition, we advise setting the `tuned-adm` profile to `hpc-compute` to ensure maximum performance from Rocky Linux.

## References

- [Futex Testing from the Linux Kernel Docs Team](https://www.kernel.org/doc/readme/tools-testing-selftests-futex-README)

- [Generic Mutex Subsystem from the Linux Kernel Docs Team](https://docs.kernel.org/locking/mutex-design.html)

- [Set the TuneD Profile via a Config File - CIQ Knowledge Base](https://kb.ciq.com/article/rocky-linux/rl-tuned-profile-set-via-config-file)


Performance Differences Between CentOS 7.9 and Rocky Linux 8.9 - A Root Cause Analysis and Recommendations


## Introduction

`cgroups v2` is an excellent evolution over `cgroups v1` and provides additional functionality such as enhanced resource allocation management, resource isolation, Pressure Stall Information, and more.

When enabling `cgroups v2` on Rocky Linux 8 however (which uses `cgroups v1` by default) for use with `podman`, there are areas of the system including your `containers.conf` and `pam` configurations, that need to be checked and updated if necessary.

## Problem

You run `sudo grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="systemd.unified_cgroup_hierarchy=1"` and then reboot your Rocky Linux 8 system to enable `cgroups v2`. However, upon running a container with `podman run`, you observe this `dbus` error:

```
dbus: couldn't determine address of session bus  Failed to create bus connection: No such file or directory
```

## Symptoms

The following commands should provide output for your user. If not, that usually means the `pam` configuration is too strict on the system and is not allowing the user's `dbus` session to run:

```bash
systemctl status user@$(id -u).service
systemctl status user-$(id -u).slice
echo $XDG_RUNTIME_DIR
```

## Resolution

Your system-wide `containers.conf` configuration at `/etc/containers/containers.conf` or at the user-level at `$HOME/.config/containers/containers.conf` should contain the below line under `[engine]`:

```text
cgroup_manager = "cgroupfs"
```

In addition, check your `pam` configuration. A configuration that is too strict, can cause the user's `dbus` session to not run and thus the `couldn't determine address of session bus` errors appear when executing `podman`. Adding the line `session optional pam_systemd.so` in `/etc/pam.d/sshd` is not enough, the same line also needs to be set in `/etc/pam.d/password-auth` and `/etc/pam.d/system-auth` as well, to ensure the user's `dbus` session activates and runs without interruption. 

## Root Cause

`cgroup_manager = "cgroupfs"` not being set in either the system-level `/etc/containers/containers.conf` file or the user-level `$HOME/.config/containers/containers.conf` configuration.

`session optional pam_systemd.so` only being set in `/etc/pam.d/sshd` and not in `/etc/pam.d/password-auth` and `/etc/pam.d/system-auth` as well.

## References & related articles

[About cgroup v2 - Kubernetes Documentation](https://kubernetes.io/docs/concepts/architecture/cgroups/)
[Lab 4: Advanced System and process monitoring - Rocky Linux Documentation](https://docs.rockylinux.org/10/labs/systems_administration_II/lab4-advanced_system_process_monitoring/)
[Podman Documentation](https://docs.podman.io/en/stable/markdown/podman.1.html)
[What does pam_systemd.so do?](https://unix.stackexchange.com/questions/626665/what-does-pam-systemd-so-do)


Avoiding dbus Errors with cgroups v2 and podman on Rocky Linux 8


## Introduction

Using PXE to bootstrap nodes with an image is commonplace in many business environments and has made both deploying and managing an image across a fleet of nodes at scale trivial.

This article will go through the setup process of deploying a PXE server on Rocky Linux 9.7 and then bootstrap a fresh node with a Rocky Linux 9.7 image. The article will focus on a UEFI boot use case, as opposed to a legacy BIOS boot.

## Prerequisites

* A node with Rocky Linux 9.7 installed on it. This will be the PXE server.

* A node that has PXE support available. This will be the PXE client.

* A Rocky Linux 9.7 DVD ISO.

## Instructions

### PXE Server Setup

To set up the PXE Server please follow the steps below:

* Update all packages:

```bash
sudo dnf update -y
```

* Install `dnsmasq` to provide a DNS and DHCP server support:

```bash
sudo dnf install -y dnsmasq
```

* Set up your `dnsmasq.conf` configuration with the appropriate subnet, netmask, broadcast-address, and more:

```bash
cat << "EOF" | sudo tee /etc/dnsmasq.conf
interface=<your_ethernet_interface>,lo
#bind-interfaces
domain=<name_here-anything_is_okay>
# DHCP range-leases
dhcp-range=<your_ethernet_interface>,<start_of_dhcp_range>,<end_of_dhcp_range>,255.255.240.0,1h
# PXE
dhcp-boot=pxelinux.0,PXEserver,<your_rocky_linux_node_address>
# Gateway
dhcp-option=3,<your_gateway_address>
# DNS
dhcp-option=6,<your_dns_server_address>
server=<your_rocky_linux_node_address>
# Broadcast Address
dhcp-option=28,<your_broadcast_address>
# NTP Server
dhcp-option=42,0.0.0.0

pxe-prompt="Press F8 for menu.", 60
pxe-service=x86PC, "Install Rocky Linux 9.7 from your PXE server <your_rocky_linux_node_address>", pxelinux
enable-tftp
tftp-root=/var/lib/tftpboot
EOF
```

Example:
*Please supplement the addresses with those from your own environment*

```bash
cat << "EOF" | sudo tee /etc/dnsmasq.conf
interface=enp8s0,lo
#bind-interfaces
domain=test
# DHCP range-leases
dhcp-range=enp8s0,10.25.96.4,10.25.96.5,255.255.240.0,1h
# PXE
dhcp-boot=pxelinux.0,pxeserver,10.25.96.3
# Gateway
dhcp-option=3,10.25.96.1
# DNS
dhcp-option=6,10.25.96.1
server=10.25.96.3
# Broadcast Address
dhcp-option=28,10.25.96.255
# NTP Server
dhcp-option=42,0.0.0.0

pxe-prompt="Press F8 for menu.", 60
pxe-service=x86PC, "Install Rocky 9.7 from network server 10.25.96.3", pxelinux
enable-tftp
tftp-root=/var/lib/tftpboot
EOF
```

*The PXE server has been assigned the address of `10.25.96.3` in the above configuration.*

* Install the `syslinux` package:

```bash
sudo dnf install -y syslinux
```

* Install the `tftp-server` package:

```bash
sudo dnf install -y tftp-server
```

* Copy the contents of the `syslinux` directory to `/var/lib/tftpboot`:

```bash
sudo cp -r /usr/share/syslinux/* /var/lib/tftpboot
```

* Create a directory called `pxelinux.cfg`:

```bash
sudo mkdir /var/lib/tftpboot/pxelinux.cfg
```

* Generate the PXE BOOT menu:

```bash
cat << "EOF" | sudo tee /var/lib/tftpboot/pxelinux.cfg/default
default menu.c32
prompt 0
timeout 300
ONTIMEOUT local

menu title # Rocky Linux 9.7 Boot Menu #

label 1
menu label ^1) Install Rocky Linux 9.7 x64 using a Remote Repository
  kernel rocky97/vmlinuz
  append initrd=rocky97/initrd.img inst.zram=1 inst.repo=https://dl.rockylinux.org/pub/rocky/9.7/BaseOS/x86_64/os/
EOF
```

**NOTE**
The PXE BOOT menu can be further expanded to include `kickstart` file configurations and other customizations. An example entry using a `kickstart` file is below:

```bash
label 2
menu label ^2) Install Rocky Linux 9.7 x64 using a Remote Repository
  kernel rocky97/vmlinuz
  append initrd=rocky97/initrd.img inst.debug ip=dhcp inst.ks=nfs:<your_rocky_linux_node_ip>:</path/to/your/kickstarter_cfg_file> inst.repo=nfs:<your_rocky_linux_node_ip>:</path/to/your/iso> inst.zram=1
```

* Create a directory called `rocky97`:

```bash
sudo mkdir /var/lib/tftpboot/rocky97
```

* Mount the Rocky Linux 9.7 DVD ISO image:

```bash
sudo mount -o loop </path/to/iso> </iso/mount_point>
```

* For this example we mounted the Rocky Linux 9.7 ISO at `/mnt`:

```bash
sudo mount -o loop ~/isos/Rocky-9.7-x86_64-dvd.iso /mnt
```

* Copy the required `initrd.img` and `vmlinuz` files to `/var/lib/tftpboot/rocky97`:

```bash
sudo cp /mnt/images/pxeboot/vmlinuz /var/lib/tftpboot/rocky97
sudo cp /mnt/images/pxeboot/initrd.img /var/lib/tftpboot/rocky97
```

* Start and enable the `dnsmasq` and `tftp` services at boot:

```bash
sudo systemctl enable --now dnsmasq
sudo systemctl enable --now tftp
```

* Allow `dhcp`, `dns`, `proxydhcp` and `tftp` traffic through your firewall:

```bash
sudo firewall-cmd --add-service=dhcp --permanent
sudo firewall-cmd --add-service=dns --permanent
sudo firewall-cmd --add-port=69/udp --permanent
sudo firewall-cmd --add-port=4011/udp --permanent
sudo firewall-cmd --reload
```

### PXE Client Setup

To set up the PXE Client please follow the steps below to enable the available PXE option from the client machine's BIOS settings. In this example, enable the following:

* Navigate to `Advanced` --> `Onboard Devices Configuration` --> `Realtek LAN Controller` and set this option to `Enabled`.

* Navigate to `Advanced` --> `Onboard Devices Configuration` --> `Realtek LAN Controller` --> `Realtek PXE Option ROM` and set this option to `Enabled`.

* Navigate to `Advanced` --> `Network Stack Configuration` --> `Network Stack` and also set this option as `Enabled`.

* Navigate to `Advanced` --> `Network Stack Configuration` --> `Ipv4 PXE Support` and set this option to `Enabled`.

* Restart your PXE client machine and it should connect to your PXE server.

* Press `F8` at the prompt:

![The prompt to press F8 shows that the PXE client machine has connected to the PXE server successfully. Press F8 to continue.](/images/articles/pxe_server_boot_1.webp)

* Press the `Enter` key at the next screen and proceed to the Rocky Linux 9.7 Boot Menu. Select `option 1`:

![The Rocky Linux 9.7 PXE Boot Menu appears. One option called "Install Rocky Linux 9.7 x64 using a Remote Repository" is present. The user is prompted to select that option.](/images/articles/pxe_server_boot_2.webp)

Your PXE client will now boot up and download the `initrd.img` and `vmlinuz` files, as well as the files from the remote repository. The Anaconda installer will then appear and you can continue the installation as normal. Of course, if you submit a `kickstart` configuration file, the installation part of the process can be skipped.

## Conclusion

This guide was built in the hope of helping bring up a simple PXE server. It touched on configurations for `dnsmasq`, `tftp`, setting up the PXE server boot menu, and more. PXE is an amazing technology and has truly revolutionized large scale deployments.


How to Setup a PXE Server on Rocky Linux 9.x


## Introduction

When provisioning Rocky Linux via PXE using NFS, you can't host both the installer source and your kickstart file on the same NFS share. This article explains how to work around this limitation by creating two separate exports, one for the kickstart file and one for the installation media.

## Problem

Attempting to point both `inst.ks` and `inst.repo` to the same NFS share results in a conflict. The installer fails to locate one of the resources, causing the automated installation to fail. For example, hosting your kickstart in `nfs:10.1.0.15:/mnt/repo/ks.cfg` and your source at `nfs:10.1.0.15:/mnt/repo/` doesn't work. As a result, you may encounter *access denied* or *failed to mount* errors.

## Resolution

### Prerequisites

`sudo` privileges are required.

### NFS configuration

On your NFS server, create two directories:

- `/mnt/ks`: for the kickstart file  
- `/mnt/repo`: for the Rocky Linux ISO or repository

Configure exports in `/etc/exports`:

```bash
/mnt/ks    10.1.0.0/24(ro,sync,no_root_squash)
/mnt/repo   10.1.0.0/24(ro,sync,no_root_squash)
```

Apply with:

```bash
exportfs -r
```

Place your kickstart file at `/mnt/ks/ks.cfg` and your ISO or repo under `/mnt/repo`.

### PXE configuration

Update your PXE menu (GRUB/iPXE) entry:

```text
menuentry 'Install Rocky Linux' --class fedora --class gnu-linux --class gnu --class os {
   linuxefi /pxeboot/vmlinuz inst.ks=nfs:10.1.0.15:/mnt/ks/ks.cfg inst.repo=nfs:10.1.0.15:/mnt/repo/ ip=dhcp
   initrdefi /pxeboot/initrd.img
}
```

## Notes

The steps listed in this article apply equally to Rocky Linux 8, 9, and 10.

## References & related articles

[Rocky Linux NFS Server Guide](https://docs.rockylinux.org/guides/file_sharing/nfsserver/)  
[How to Setup a PXE Server on Rocky Linux 9.x](https://kb.ciq.com/article/rocky-linux/rl-pxe-boot-kickstart-file)


PXE Install Fails on Rocky Linux When Kickstart and Repo Share NFS Export


## Introduction

This article addresses an issue surrounding the `python3-devel` package in Rocky Linux 8.10, after upgrading from 8.9. It provides guidance on proper package usage during development and package building processes.

## Problem

When upgrading from Rocky Linux 8.9 to Rocky Linux 8.10, if the `python3-devel` package was previously installed, the user may see the following error after running the `dnf update` command:

```bash
Error:
 Problem: cannot install both platform-python-3.6.8-62.el8_10.rocky.0.x86_64 from baseos and platform-python-3.6.8-56.el8_9.3.rocky.0.x86_64 from @System
  - package python3-devel-3.6.8-56.el8_9.3.rocky.0.x86_64 from @System requires platform-python = 3.6.8-56.el8_9.3.rocky.0, but none of the providers can be installed
  - cannot install the best update candidate for package platform-python-3.6.8-56.el8_9.3.rocky.0.x86_64
  - problem with installed package python3-devel-3.6.8-56.el8_9.3.rocky.0.x86_64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
```

## Resolution

1.- Disable the `devel` repository unless actively building packages. The `devel` repository is intended only for development/build environments. It may contain packages not included in the `base` or `extras` repositories, and is not recommended for general usage.

2.- Use the `platform-python-devel` package instead of `python3-devel`. On RHEL, `python3-devel` is not available. Instead, the appropriate development package is `platform-python-devel`, which is included in the `AppStream` repository.

you can use the `--allowerasing` option to move forward with the update or manually remove the `python3-devel` package and install `platform-python-devel` by running:

```bash
dnf install platform-python-devel
```

## Notes

Use recommended macros and alternatives for compatibility. You can check the package info by running:

```bash
rpm -qpi https://download.rockylinux.org/pub/rocky/8/devel/x86_64/os/Packages/p/python3-devel-3.6.8-62.el8_10.rocky.0.x86_64.rpm
```

The `python3-devel` package specifically states:

```bash
It also makes the "python3" and "python3-config" commands available for compatibility with some build systems.
When building packages, prefer requiring platform-python-devel and using
the %{__python3} macro instead, if possible.
```

If the python3 command is needed in this specific case, consider installing the `python36` packages instead.

## References & related articles

[Rocky Linux Documentation, Notes on: Devel](https://wiki.rockylinux.org/rocky/repo/#notes-on-devel)


Python Devel Package Not Available for Rocky Linux 8.10


## Introduction

In high-availability server environments, minimizing downtime is critical. Renaming a network interface often requires a reboot, which can disrupt services. This guide provides methods to rename network interfaces dynamically, ensuring changes take effect without requiring a system restart.

## Problem

Renaming network interfaces without rebooting can be a challenge in Linux systems. Traditional methods typically require a restart to apply changes, which isn't ideal for systems that demand continuous uptime. This limitation can hinder administrators managing critical services or performing live configurations.

## Resolution

### Prerequisites

Access to the `root` user account or a user with escalated privileges.

### Method 1 - `ip link set`

To temporarily change the name of an interface, use the `ip link set` command.

Bring down the network interface that you wish to rename:

```bash
sudo ip link set <INTERFACE_NAME> down
```

Rename the interface:

```bash
sudo ip link set <INTERFACE_NAME> name <NEW_INTERFACE_NAME>
```

Enable the newly renamed interface:

```bash
sudo ip link set <NEW_INTERFACE_NAME> up
```

**Please note that rebooting your server removes the changes.**

### Method 2 - `udev` rules by MAC address

Check under `/etc/udev/rules.d/` that you have a `70-persistent-net.rules` file. If the file isn't there, create it with `touch /etc/udev/rules.d/70-persistent-net.rules`

Confirm the interface you wish to edit by using `ip -brief address show`. You can see a similar output as in the below example:

```bash
$ ip -brief address show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
enp8s0           UP             <PUBLIC_IP>/24
```

Open `/etc/udev/rules.d/70-persistent-net.rules` with your text editor of choice and modify the `NAME` field of the interface you wish to change. A before and after example follows:

Before:

```bash
$ cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="<MAC_ADDRESS_HERE>", NAME="enp8s0"
```

After:

```bash
$ cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="<MAC_ADDRESS_HERE>", NAME="new-name-1"
```

Bring the network interfaces down with `ip link set <INTERFACE_NAME> down`. In this case `enp8s0`:

```bash
ip link set enp8s0 down 
```

Use the `udevadm control` command to reload the `udev` rules and request device events from the kernel with `udevadm trigger`:

```bash
udevadm control --reload-rules; udevadm trigger --subsystem-match=net --action=add
```

Use `ip -brief address show` to confirm that the system has renamed the network interface:

```bash
$ ip -brief address show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
new-name-1       DOWN           <PUBLIC_IP>/24
```

Bring the link back up using `ip link set <NEW_INTERFACE_NAME> up`. Please see the following example:

```bash
ip link set new-name-1 up 
```

Observe that the system renamed the interface and brought it up without requiring a reboot:

```bash
$ ip -brief address show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
new-name-1       UP             <PUBLIC_IP>/24
```

Even if you reboot the machine, the changes remain persistent.

## References & related articles

[`udevadm` man page](https://linux.die.net/man/8/udevadm)  
[`udev` rules introduction](https://opensource.com/article/18/11/udev)


Renaming Network Interfaces Without Rebooting


## Introduction

Losing access to the root account on a Rocky Linux server can be an issue when you need to perform administrative tasks or troubleshooting. You can regain root access by resetting the root password using the GRUB bootloader. This article will guide you through the process, which works for Rocky Linux 8, 9, and 10.

## Resolution

There are two primary methods to reset the root password: using `init=/bin/sh` or `rd.break` at boot time. Both are effective, but `init=/bin/sh` is generally more straightforward and works more reliably across Rocky Linux 8, 9, and 10, including virtual and physical machines. The `rd.break` method is often preferred as well and works in situations where you need to perform more extensive operations within the initial ramdisk environment before the root filesystem is fully mounted, such as dealing with complex LVM configurations or troubleshooting boot time issues.

### Using `init=/bin/sh`

This method is recommended because it is simple, does not require chrooting, and works consistently across most Rocky Linux installations:

- Reboot the server.
- At the GRUB menu, highlight the kernel you wish to boot and press `e` to edit.
- Find the line starting with `linux` or `linux16` or `linuxefi`.
- At the end of this line, add: `init=/bin/sh`
- Press `Ctrl + x` to boot with these parameters.

![GRUB menu with init=/bin/sh highlighted; the target line is the fourth one starting with "linux".](/images/articles/reset-root-password/grub-screen.jpg)

- When the shell prompt appears, remount the root filesystem as read-write:

  ```shell
  mount -o remount,rw /
  ```

- Reset the root password:

  ```shell
  passwd root
  ```

- If SELinux is enabled, ensure proper relabeling on next boot:

  ```shell
  touch /.autorelabel
  ```

- Reboot the system:

  ```shell
  /usr/sbin/reboot -f
  ```

### Using `rd.break`

This method serves as a helpful alternative if you experience difficulties with the primary approach, or if your system uses LVM or a more complex partitioning scheme.

- Reboot the server.
- At the GRUB menu, highlight the kernel and press `e` to edit.
- On the line starting with `linux`, add `rd.break` to the end.
- Press `Ctrl + x` to boot.
- At the `switch_root` prompt, remount the sysroot as read-write:

  ```shell
  mount -o remount,rw /sysroot
  ```

- Change root into the sysroot:

  ```shell
  chroot /sysroot
  ```

- Reset the root password:

  ```shell
  passwd root
  ```

- Relabel for SELinux:

  ```shell
  touch /.autorelabel
  ```

- Exit the chroot and reboot:

  ```shell
  exit
  exit
  ```

## Notes

- Console or vconsole directives in the GRUB boot line (e.g., `console=ttyS0,115200n8`) may need to be removed if you are working in a virtual machine or if you do not see the expected shell prompt.
- If SELinux is enabled and you do not relabel the filesystem, you may encounter login issues after resetting the password.
- The relabel process can take anywhere from a few minutes to over half an hour, depending on your system.
- These procedures require physical or virtual console access to the server.

## References & related articles

[Rocky Linux System Startup Guide](https://docs.rockylinux.org/books/admin_guide/10-boot/)  
[Rocky Linux User Management Guide](https://docs.rockylinux.org/books/admin_guide/06-users/)


How to Reset a Lost Root Password on Rocky Linux


## Introduction

This article addresses the issue where the Sriov configuration script runs manually but fails when executed automatically by systemd. The root cause is identified as a timing or dependency issue.

## Problem

The script `/etc/sriov/setup_dpdk_nic.sh` works when executed manually after a node restart but fails when run automatically by systemd. This indicates a potential timing or dependency issue with the service startup.

## Symptoms

- Script executes successfully when run manually after node restart.
- Script fails when triggered automatically by systemd during the boot process.

## Resolution

Adjust the service unit file to address potential timing and dependency issues. Update the service unit file as follows:

```ini
[Unit]
Description=Sriov Config Service
After=network-online.target systemd-udevd.service
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/etc/sriov/setup_dpdk_nic.sh
RemainAfterExit=yes
TimeoutStopSec=90
#StartLimitInterval=120
#StartLimitBurst=3

[Install]
WantedBy=multi-user.target
```

### Explanation of Changes

1. **[Unit] Section:**
   - **After=network-online.target systemd-udevd.service:** Ensures the service starts only after the network is fully online and systemd-udevd is running. This is crucial for NIC-related configurations.
   - **Wants=network-online.target:** Ensures the network is available when the service runs.

2. **[Service] Section:**
   - **RemainAfterExit=yes:** Keeps the service active after the script completes, aiding in tracking and status reporting.
   - **TimeoutStopSec=90:** Sets a timeout for the stop operation.
   - **StartLimit* Parameters:** Commented out as they control service restart limits, which are less relevant for a oneshot service.

### Benefits

- Ensures the NIC configuration script runs at the correct time during the boot process.
- Improves service status reporting by keeping the service active after script execution.

### Troubleshooting

If the issue persists, provide the output of the following command for further review:

```bash
journalctl -u sriov-configure.service
```

## Root Cause

The problem likely stemmed from the script running before the network was fully online. By adjusting the service dependencies, we ensure the script runs at the appropriate time during the boot sequence.


Resolving Timing and Dependency Issues in Systemd Service Unit for Sriov Configuration


## Introduction

Dual-booting is the practice of installing and using two separate operating systems on one computer. This setup enables you to select which operating system to boot into during startup, offering the flexibility to switch between them as needed.

## Problem

After installing the August 2024 Windows security update, [KB5041585](https://support.microsoft.com/help/5043076) or the August 2024 preview update, you might face issues with booting Linux if you have enabled a dual-boot setup of Windows and Linux on your machine. The result of this issue, is your device might fail to boot Linux and show the following error message:

```shell
Verifying shim SBAT data failed: Security Policy Violation
Something has gone seriously wrong: SBAT self-check failed: Security Policy Violation
```

## Resolution

Please follow the instructions highlighted in [this Microsoft article](https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-23h2#august-2024).

## Root Cause

The August 2024 Windows security and preview updates apply a Secure Boot Advanced Targeting (SBAT) setting to machines that run Windows. This blocks old and vulnerable boot managers. This SBAT update will not be applied to devices where dual-booting is detected. On some machines, the dual-boot detection did not detect some customized methods of dual-booting and applied the SBAT value when it shouldn't have done so.

---
**_IMPORTANT:_** _This known issue only occurs with the installation of the August 2024 security and preview updates. The September 2024 [KB5043076](https://support.microsoft.com/en-us/topic/september-10-2024-kb5043076-os-builds-22621-4169-and-22631-4169-215aad1e-3f3f-44bd-9868-91a2bd450a07) and later security updates do not contain the settings that cause this issue. If you install the September 2024 update, you don’t need to apply the workaround listed above._

---

## Notes

- Disabling Secure Boot is a temporary workaround, but is not recommended for long-term security.
- Ensure your system firmware (BIOS/UEFI) is up to date before applying updates.
- Monitor Windows Update for any future patches addressing this issue.

## References & Related Articles (Optional)

- [Microsoft August 2024 Update Support Article](https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-23h2#august-2024)


Rocky Linux Fails to Boot in a Dual-Boot Setup with Windows


## Introduction

This article provides step-by-step guidance on how to modify screen lock settings in the XFCE desktop environment on Rocky Linux. Whether you need to disable the screen lock for remote sessions or enforce a 10 minute inactivity timeout for local logins, this document offers both GUI and CLI-based solutions. The methods discussed here are aimed at helping you lock or unlock specific settings, ensuring consistency and security across different session types, whether that be local or via a remote session.

## Problem

There are often challenges in managing screen lock preferences, especially when different environments require conflicting behaviors. In some instances, remote sessions should have the lock disabled to avoid disruptions, while local sessions must enforce a 10 minute timeout for security.

## Symptoms

- The screen lock activates automatically at inappropriate times.

- Changes via the GUI or `xfce4-settings-editor` are not immediately apparent, leading to user confusion.

## Resolution

### Prerequisites

- Rocky Linux 8.10 installed on a test machine.

- A fully updated system.

- XFCE desktop installed and properly configured.

- (Optional) NoMachine configured for remote access testing.

### Screenlock configuration methods

#### GUI method

##### Power Manager – System

- Open **Settings → Power Manager → System**.

- Set the **When inactive for** option to **Never** to disable automatic screen locking.

- Ensure that under **Security**, the **Lock screen when system is going to sleep** option is unchecked.

##### Power Manager – Display:

- Open **Settings → Power Manager → Display**.

- Confirm that **Display power management** is set to ON.

- Set **Blank after**, **Put to sleep after**, and **Switch off after** to **Never**.

##### Screensaver Settings:

- Open **Settings → Screensaver**.

- Disable the **Enable Screensaver** option.

- Under **Lock Screen**, uncheck the **Enable Lock Screen** option.

#### CLI method

##### To disable the automatic screen lock for the session:

```bash
xfconf-query -c xfce4-session -p /general/LockCommand -s ""
```

##### To re-enable the lock screen:

```bash
xfconf-query -c xfce4-session -p /general/LockCommand -s "/usr/bin/gnome-screensaver-command --lock"
```

##### Toggle the power management (DPMS-Display Power Management Signaling-controls whether screen blanking runs or not) setting on/off:

```bash
xfconf-query -c xfce4-power-manager -p /xfce4-power-manager/dpms-enabled -T
```

##### Toggle the screen lock on/off:

```bash
xfconf-query -c xfce4-screensaver -p /lock/enabled -T
```

##### Toggle the screensaver on/off:

```bash
xfconf-query -c xfce4-screensaver -p /saver/enabled -T
```

##### To disable the lock screen:

```bash
xfconf-query -c xfce4-screensaver -p /lock/enabled -s false
```

##### To enable the lock screen:

```bash
xfconf-query -c xfce4-screensaver -p /lock/enabled -s true
```

##### Add a 10 minute delay before the screen locks:

```bash
xfconf-query -c xfce4-screensaver -p /lock/saver-activation/delay -s 10
```

##### How to verbosely list settings:

```bash
xfconf-query -c xfce4-power-manager -l -v
xfconf-query -c xfce4-screensaver -l -v
xfconf-query -c xfce4-session -l -v
```

## Root cause

The challenges encountered stem from XFCE’s management of screen lock settings across multiple independent components (Power Manager, Screensaver, and Session settings). These components can lead to conflicting behaviors when trying to restrict or enforce specific lock preferences, especially in environments that require different configurations for remote versus local sessions.

## Notes

Always test changes in a controlled or non-production environment before deployment.

**Tip:** Keep the `xfce4-settings-editor` open during any changes to observe the settings being updated in real-time.

## References & related articles

[XFCE Power Manager Documentation](https://docs.xfce.org/xfce/xfce4-power-manager/start)  
[XFCE Screensaver Configuration Documentation](https://docs.xfce.org/apps/xfce4-screensaver/start)  
[NoMachine for Linux](https://www.nomachine.com)


Managing Screen Lock Preferences in XFCE on Rocky Linux


## Introduction

PyKdump is a powerful extension for the `crash utility`, designed to integrate Python scripting into the crash analysis workflow. It enhances the `crash utility`'s capabilities by providing additional commands and tools that streamline the investigation of `vmcore` dumps.

In particular, PyKdump provides invaluable insight when delving into SCSI device details. Its commands, such as `scsishow --check`, provide additional information on SCSI command statuses, error codes, and device interactions that are not available in the standard `crash utility`.

This article details the installation of the PyKdump extension and recommended SCSI commands to help analyze a `vmcore` dump for a node's filesystem that has reached full capacity.

## Problem

Kernel crashes can occur when a disk is filled to 100%, resulting in a prolonged process halt and eventual panic.

An `sosreport` depending on the time it is taken, sometimes does not reveal filesystem overutilization. CIQ highly recommends to take the `sosreport` as soon as the issue occurs.

If you are unsure as to how to generate an `sosreport`, [please see the full guide available here.](https://kb.ciq.com/article/rocky-linux/rl-generate-an-sosreport)

If the issue cannot be identified in the `sosreport`,  this necessitates further analysis of the `vmcore` dump to pinpoint filesystem-related issues.

## Symptoms

* The system experiences a complete halt of processes for a period of time before the kernel panic ensues.

* Despite the severe impact, the post-crash `sosreport` does not show any evidence of a filesystem filling up, making the root cause from the stance of the `sosreport` difficult to track down.

## Resolution

### Prerequisites
  
* Rocky Linux 8.10

* This article assumes you are familiar with setting up and using the crash utility. For more information, please follow [this guide](https://kb.ciq.com/article/rocky-linux/rl-analysis-of-vmcore-dump-with-crash).

### PyKdump installation

* You will need to install `Python 3.8.20` in order for PyKdump to compile successfully.

* In addition, you also need to build `crash 8.0.5` from source for PyKdump to work.

* Install the `Development Tools` group of packages:

```bash
dnf group install -y "Development Tools"
```

* Please run the below BASH script in order to compile `Python 3.8.20`, `crash` and build `PyKdump`:

```bash
#!/bin/bash

dnf config-manager --set-enabled powertools

dnf install -y readline-devel

dnf install -y texinfo

dnf install -y lzo-devel

git clone git://git.code.sf.net/p/pykdump/code pykdump

wget https://www.python.org/ftp/python/3.8.20/Python-3.8.20.tar.xz

tar -xf ./Python-3.8.20.tar.xz

cd Python-3.8.20/

./configure CFLAGS=-fPIC --disable-shared

cp ../pykdump/Extension/Setup.local-3.8 ./Modules/Setup.local

make

cd ..

wget https://github.com/crash-utility/crash/archive/refs/tags/8.0.5.tar.gz

tar -xf 8.0.5.tar.gz

cd crash-8.0.5/

make lzo

cd ~/pykdump/Extension

./configure -p /root/Python-3.8.20 -c /root/crash-8.0.5

make

cp mpykdump.so ~/mpykdump.so
```

### Pkdump usage

* Move the `crash` binary over to `/usr/bin`:

```bash
sudo cp /path/to/crash-8.0.5 /usr/bin
```

* Start the `crash utility` with a `vmlinux` file and `vmcore` dump selected using this command:

```bash
crash /path/to/vmlinux /path/to/vmcore
```

* Enable the PyKdump extension by running the following in the `crash utility` CLI:

```bash
crash> extend /root/mpykdump.so
Setting scroll off while initializing PyKdump
/root/mpykdump.so: shared object loaded
```

* Here are some recommended commands with which to find more information about `scsi` devices:

### Recommended commands

#### To get a summary of all of your SCSI devices

```bash
scsishow --check

### Summary:

    Task                             Errors/Warnings
    ------------------------------------------------
    SCSI host checks:                0
    SCSI device, command checks:     1
    SCSI target checks:              0

 ** Execution took   0.05s (real)   0.05s (CPU)
```

#### To get the I/O requests pending for a device

```bash
scsishow -r

fa41a307e427e3s0 (10:0:0:1)    timeout: 30000      deadline: 11590928508
Requests found in SCSI layer: 1

 ** Execution took   0.04s (real)   0.04s (CPU)
```

#### To get the last in-flight SCSI commands that were running at the time of the crash

```bash
scsishow -c

scsi_cmnd ff41a507e427e4f8 on scsi_device 0xffa41a307e427e3s0 (10:0:0:1) jiffies_at_alloc: 11590898508
```

#### To check SCSI host adapters for any that are in `busy`, `blocker` or `failed` state

```bash
scsishow -s | grep host

NAME      NAME                   Scsi_Host                shost_data               hostdata                
host0     ahci                   ffa41a307e427e3s0                       0         ff21b221c1429b58
   host_busy           : 0
   host_blocked        : 0
   host_failed         : 0
   host_self_blocked   : 0
   shost_state         : SHOST_RUNNING
```

#### To check for high `IOERR-CNT` I/O error counts

```bash
scsishow -d | awk '{print $1,$NF}' | sort -nrk2

sda 3712821
```

#### To get the `IOREQ-CNT` (I/O Request Count) and the `IODONE-CNT` (I/O Done Count)

```bash
scsishow -d | awk '{print $1,$8,$9}' | sort -nrk2

device / IOREQ-CNT / IODONE-CNT
sdw 10500750 10500750
```

## References & related articles

[PyKdump User Documentation](https://pykdump.readthedocs.io/en/latest/userdoc/index.html)  
[Linux Kernel Crash Book by Igor Ljubuncic](https://www.dedoimedo.com/computers/www.dedoimedo.com-crash-book.pdf)


How to Perform Further SCSI Device Analysis with the PyKdump Crash Extension


## Introduction

Smart cards add an additional layer of authentication to your Linux system. Smart cards have the benefit of being a physical device, that (generally speaking with some exceptions) cannot be reprogrammed once the private keys have been inserted. 

This article will go over how to setup smart card authentication on the MATE desktop environment. The article assumes that the user has a Yubikey / other form of smart card available and has knowledge of setting up a smart card on GNOME. The following method requires the installation of the GNOME desktop environment.

## Problem

You are running Rocky Linux in an enterprise environment, with workstations that have the MATE desktop environment installed on them. You want to add an additional layer of security, by making sure all employees log in using their smart card.

## Resolution

Set up a Rocky Linux 9 System with the MATE desktop. The pre-built [Desktop/Workstation Live Image](https://dl.rockylinux.org/pub/rocky/9/live/x86_64/Rocky-9-MATE-x86_64-latest.iso) is a good option for a smooth install experience.

The GDM lock screen from the GNOME desktop environment is needed for the smart card authentication process. The easiest way to obtain `GDM` (and the GNOME desktop environment as well) is to install the `Workstation` group package with:

```bash
dnf group install "Workstation"
```

* Then enable the GDM service with:

```bash
systemctl enable gdm --force
```

* After that, reboot the machine and the GDM login manager will appear.

* Log into GNOME and configure your smart card.

* Once done, log back out to the GDM login manager.

* Click the little cog icon in the bottom right-hand corner, select the `MATE` option, enter your password, and continue to log in from there.

* To lock your session, run the command below. This will switch the current `MATE` session to the GDM lock screen:

```bash
gdmflexiserver -ls
```

## References & related articles

[Ubuntu MATE Forums - Setup and Locking/Unlocking MATE With a Smart Card](https://ubuntu-mate.community/t/setup-and-locking-unlocking-mate-with-a-smart-card/27461/19)


Enabling Smart Card Support on the MATE Desktop Environment


## Introduction

This article addresses soft lockup issues that can occur during application execution on Rocky Linux.

The recommendations provided here are based on a Slurm environment with high CPU utilization from a Python application, causing CPU lockups.

This guide focuses on Rocky Linux 8.x and 9.x.

## Problem

Soft lockup messages are observed from an application with high CPU utilization.

## Symptoms

System logs from `dmesg` display lockup messages such as `BUG: soft lockup - CPU#18 stuck for 20s! [python3:104938]` during program execution.

`dmesg` also includes `raw_spin_unlock_irqrestore` messages, indicating that a process may be holding a lock for an extended period and causing contention of system resources.

The timing of the soft lockup messages are closely tied to the `watchdog threshold` setting, which can be checked under `/proc/sys/kernel/watchdog_thresh`.

## Resolution

### Improperly configured `/etc/sysconfig/grub` file

- Verify that the `/etc/sysconfig/grub` file is configured correctly with options such as `console=ttyS1` or `console=ttyS1,9600`, and consider increasing the baud rate to `115200` or disabling serial console logging.

### High load average

- High load averages, often exceeding the number of available logical CPU cores, can cause further issues.

- It is worth installing the `sysstat` package with `sudo dnf install -y sysstat` and checking the generated `/var/log/sa` files.

- If the load average is higher than the total number of logical CPU cores available, that can also cause the `soft lockup` messages.

### The watchdog_thresh kernel parameter configuration

- The value set in `watchdog_thresh` determines the threshold interval the kernel uses to decide when a CPU is “stuck.”

- By default, the `watchdog_thresh` value is set to `10` seconds.

- The soft lockup threshold is twice the value of the `watchdog_thresh` parameter.

- The general advice is not to change the default `watchdog_thresh` value, as this can mask hard lockups that occur.

- Verify the current `watchdog_thresh` value with the following command:

```bash
cat /proc/sys/kernel/watchdog_thresh
```

### Application tuning

- Investigate the behavior of the application to identify if it is monopolizing CPU resources, and tweak it to reduce CPU load.

### Tuned profile selection

- Apply the `tuned` profile optimized for high-performance computing workloads by running:

```bash
sudo dnf install -y tuned
sudo systemctl enable --now tuned
sudo tuned-adm profile hpc-compute
sudo tuned-adm active
```

### CPU time limitations

- Limit CPU time for user processes by updating `/etc/security/limits.conf`. An example is below:

```bash
username soft cpu 60  
username hard cpu 120
```

- In the above example, the `soft` limit (this can be moved up or down by the user within the `hard` limit) is `60` seconds and the `hard` limit is enforced by the kernel and is set to `120` seconds. The `hard` limit cannot be increased by the user.

### Vmcore dump analysis

- While configuring `kdump` is outside the scope of this article, once configured, trigger a crash using the following commands:

```bash
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger
```

- By default, the `vmcore` dump will be available under `/var/crash`.

- Going through the `vmcore` dump using the `crash` utility, will also help you understand what is causing the softlockup issues.

## Root cause

Soft lockups occurs when the kernel scheduler is starved and cannot perform its duties within the allotted watchdog threshold.

A high system workload, particularly from a misbehaving application, can saturate CPU resources and prevent task switching from occurring.

System misconfigurations, such as improper serial console settings or not setting the correct `tuned` profile, can also cause lockups to occur.

## References & related articles

[limits.conf man page](https://linux.die.net/man/5/limits.conf)  
[Linux limits.conf man page](https://man7.org/linux/man-pages/man5/limits.conf.5.html)  
[Kernel Crash Dump documentation](https://www.kernel.org/doc/Documentation/kdump/kdump.txt)  
[watchdog_thresh documentation](https://www.kernel.org/doc/html/latest/admin-guide/sysctl/kernel.html#watchdog-thresh)


Diagnosing and Mitigating Soft Lockup Issues on Rocky Linux


## Introduction

When creating ext4 filesystems on loopback devices (image files), you may encounter unexpected behavior where a fully allocated image file becomes sparse after mounting. This article explains why this happens and how to prevent it.

## Problem

When creating an ext4 filesystem on a fully allocated file and then mounting it, some blocks may be unexpectedly optimized or trimmed, potentially resulting in a sparse file. This behavior can be problematic in situations where you need the backing file to remain fully allocated to ensure the mounted filesystem stays robust against ENOSPC (no space left on device) errors on the host filesystem.

## Symptoms

The issue can be observed through the following sequence:

1. Create an allocated file of roughly 5 GB:

   ```bash
   fallocate -l 5368709120 cache.img
   ```

2. Confirm the allocation:

   ```bash
   stat "--printf=%b\n" cache.img
   # Output: 10485784
   ```

3. Create an ext4 filesystem:

   ```bash
   mkfs.ext4 -E nodiscard cache.img
   ```

4. Check allocation again:

   ```bash
   stat "--printf=%b\n" cache.img
   # Output: 10485784
   ```

5. Mount the filesystem:

   ```bash
   mount -o defaults,nodiscard cache.img /mnt/test
   ```

6. After a short period, check allocation:

   ```bash
   stat "--printf=%b\n" cache.img
   # Output: 10321936
   ```

Notice that the number of allocated blocks may decrease after mounting, indicating that the file has become sparse despite using the `nodiscard` option.

## Resolution

To prevent ext4 from sparsifying the loopback file during the first mount, use the following options when creating the filesystem:

```bash
mkfs.ext4 -E nodiscard,lazy_itable_init=0,lazy_journal_init=0 cache.img
```

These options disable the lazy initialization features that defer part of the filesystem creation to the kernel at first mount time, which can cause the sparsification.

## Root Cause

The issue can occur because ext4's default behavior includes lazy initialization of the inode table and journal. When using the default settings:

1. `mkfs.ext4` creates the filesystem but defers some initialization work
2. During the first mount, the kernel completes the initialization
3. As part of this process, certain I/O patterns or optimizations may lead to blocks being treated as sparse, depending on the host filesystem and configuration

The `lazy_itable_init=0` and `lazy_journal_init=0` options force more complete initialization during filesystem creation, helping to prevent deferred work that could result in sparsification.

## Notes

- This behavior is specific to ext4 filesystems. Other filesystems like XFS do not typically exhibit this behavior.
- The `nodiscard` mount option alone may be insufficient to prevent this behavior in all cases.
- While disabling lazy initialization can prevent sparsification, it may increase the time required to create the filesystem.

## References

[Rocky Linux Filesystem Guide](https://docs.rockylinux.org/books/admin_guide/07-file-systems/)  
[ext4 mkfs documentation](https://man7.org/linux/man-pages/man8/mke2fs.8.html)  
[ext4 mount options](https://man7.org/linux/man-pages/man5/ext4.5.html)


Preventing Sparse Allocation in ext4 Loopback Filesystems


## Introduction

To get the most optimizations and power from your hardware in an Enterprise Linux system, utilization of `TuneD` and `tuned-adm` is paramount.

This guide will explore setting your `tuned` profiles via a configuration file. If you are a System Administrator that deploys pre-configured Rocky Linux images across a fleet of machines, the steps listed here will help you set a pre-configured `tuned` profile.

## Problem

A `tuned profile` can be set via the CLI, for example, with `tuned-adm profile hpc-compute`, however, you're looking to set the profile via a configuration file.

## Resolution

Designate the profile that you want set on the system. In this example, we'll be setting `hpc-compute` as the default profile, as we want to fully utilize the available system resources.

By default, the `tuned` profile that is selected when installing Rocky Linux as a virtual machine is `virtual-guest`. That is the default across Rocky Linux 8, 9, and 10.

For bare metal Rocky Linux installations, the default `tuned` profile across Rocky Linux 8, 9, and 10 is `balanced`.

To set a `tuned` profile in a configuration file, you need to use the `child profile` system. The main profiles are located in `/usr/lib/tuned/`, while the child profiles are located under `/etc/tuned/`. `/etc/tuned/` is also where you can customize profiles.

### Example configuration with hpc-compute

In this example, the bare metal node has `balanced` set as the default profile. We will be changing the default profile to `hpc-compute` and storing the change in a configuration file.

Create the directory:

```bash
mkdir /etc/tuned/balanced/
```

Copy the contents of `/usr/lib/tuned/hpc-compute/tuned.conf` into `/etc/tuned/balanced/tuned.conf`:

```bash
cat /usr/lib/tuned/hpc-compute/tuned.conf > /etc/tuned/balanced/tuned.conf
```

Through the use of `child profiles`, we have now transformed the default `balanced` profile into an `hpc-compute` profile.

Restart the tuned service to apply the new configuration:

```bash
systemctl restart tuned
Observe that the active profile has now been changed to `hpc-compute`:

```bash
cat /etc/tuned/balanced/tuned.conf
#
# tuned configuration
#

[main]
summary=Optimize for HPC compute workloads
description=Configures virtual memory, CPU governors, and network settings for HPC compute workloads.
include=latency-performance

[vm]
# Most HPC applications can take advantage of hugepages. Force them to on.
transparent_hugepages=always

[disk]
# Increase the readahead value to support large, contiguous, files.
readahead=>4096

[sysctl]
# Keep a reasonable amount of memory free to support large mem requests
vm.min_free_kbytes=135168

# Most HPC applications are NUMA aware. Enabling zone reclaim ensures
# memory is reclaimed and reallocated from local pages. Disabling
# automatic NUMA balancing prevents unwanted memory unmapping.
vm.zone_reclaim_mode=1
kernel.numa_balancing=0

# Busy polling helps reduce latency in the network receive path
# by allowing socket layer code to poll the receive queue of a
# network device, and disabling network interrupts.
# busy_read value greater than 0 enables busy polling. Recommended
# net.core.busy_read value is 50.
# busy_poll value greater than 0 enables polling globally.
# Recommended net.core.busy_poll value is 50.
net.core.busy_read=50
net.core.busy_poll=50

# TCP fast open reduces network latency by enabling data exchange
# during the sender's initial TCP SYN. The value 3 enables fast open
# on client and server connections.
net.ipv4.tcp_fastopen=3
```

* In the above example, running `tuned-adm active` will still show the profile as `balanced`, but in fact will be set to the `hpc-compute` configuration instead.

## References & Related Articles

[Rocky Linux tuned man page](https://rockyman.svr.pizza/8.10/tuned/man8/tuned.8.html)
[Rocky Linux tuned-adm man page](https://rockyman.svr.pizza/8.10/tuned/man8/tuned-adm.8.html)


Set the TuneD Profile via a Config File


## Introduction

[`udev`](https://linux.die.net/man/8/udev) provides a dynamic device directory containing only the files for actually present devices. It creates or removes device node files usually located in the `/dev` directory, or it renames network interfaces.

As part of the hotplug subsystem, `udev` is executed if a kernel device is added or removed from the system. On device creation, `udev` reads the `sysfs` directory of the given device to collect device attributes such as label, serial number or bus device number. These attributes may be used as keys to determine a unique name for the device.

On another hand, RAID devices are virtual devices created from two or more real block devices. This allows multiple devices (typically disk drives or partitions thereof) to be combined into a single device to hold, for example, a single filesystem. Some RAID levels include redundancy and so can survive some degree of device failure. Linux Software RAID devices are implemented through the `md` (Multiple Devices) device driver. More information can be found [at the mdadm man page.](https://linux.die.net/man/8/mdadm)

## Problem

The user may have created `udev` rules to handle the RAID device creation, causing the `md0_sync` process to hang.

## Symptoms

*Please note that the replication of this issue is random, as the race condition does not happen every time. You may have to run the below `mdam` command a few times, in order to recreate the race condition.*

The following udev rules were used:

```bash
SUBSYSTEM=="block",ACTION=="add|change",KERNEL=="md*",\
ATTR{md/sync_speed_max}="2000000",\
ATTR{md/group_thread_cnt}="64",\
ATTR{md/stripe_cache_size}="8192",\
ATTR{queue/nomerges}="2",\
ATTR{queue/nr_requests}="1023",\
ATTR{queue/rotational}="0",\
ATTR{queue/rq_affinity}="2",\
ATTR{queue/scheduler}="none",\
ATTR{queue/add_random}="0",\
ATTR{queue/max_sectors_kb}="4096"
```

Then to create the RAID device, the following command was run:

```bash
$ mdadm --create --verbose --chunk=128 --level=6 -n 4 /dev/md0 /dev/vda /dev/vdb /dev/vdc /dev/vdd
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: size set to 52395008K
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
```

You can check the status of the RAID creation by checking the `/proc/mdstat` file:

```bash
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 vdd[3] vdc[2] vdb[1] vda[0]
      104790016 blocks super 1.2 level 6, 128k chunk, algorithm 2 [4/4] [UUUU]
      [>....................]  resync =  4.2% (2203904/52395008) finish=3.7min speed=220390K/sec
```

When you check for a second time, you may find the file is empty or the `resync` completion percentage status has not moved.

You could find the following errors in the logs:

```bash
Sep 11 18:39:21 rockylinux.com systemd-udevd[3394]: nvme2n1: /etc/udev/rules.d/udev.rules:53 Failed to write ATTR{/sys/devices/pci0000:60/0000:60:03.3/0000:65:00.0/nvme/nvme2/nvme2n1/queue/max_sectors_kb}, ignoring: Invalid argument
Sep 12 10:22:33 rockylinux.com systemd-udevd[570894]: nvme2n1: /etc/udev/rules.d/udev.rules:53 Failed to write ATTR{/sys/devices/pci0000:60/0000:60:03.3/0000:65:00.0/nvme/nvme2/nvme2n1/queue/max_sectors_kb}, ignoring: Invalid argument
Sep 12 10:43:13 rockylinux.com systemd-udevd[3450]: nvme2n1: /etc/udev/rules.d/udev.rules:53 Failed to write ATTR{/sys/devices/pci0000:60/0000:60:03.3/0000:65:00.0/nvme/nvme2/nvme2n1/queue/max_sectors_kb}, ignoring: Invalid argument
Sep 12 12:21:44 rockylinux.com systemd-udevd[65370]: nvme2n1: /etc/udev/rules.d/udev.rules:53 Failed to write ATTR{/sys/devices/pci0000:60/0000:60:03.3/0000:65:00.0/nvme/nvme2/nvme2n1/queue/max_sectors_kb}, ignoring: Invalid argument
```

The `md0_resync` task will hang with the following logs (*it may cause the kernel to panic and crash, if you have `kernel.hung_task_panic = 1` included in your `/etc/sysctl.conf` file*):

```bash
Sep 10 10:40:38 rockylinux.com kernel: INFO: task md0_resync:2365161 blocked for more than 122 seconds.
Sep 10 10:40:38 rockylinux.com kernel:      Tainted: G S      W6.6.32-120240613.el9.x86_64 #1
Sep 10 10:40:38 rockylinux.com kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 10 10:40:38 rockylinux.com kernel: task:md0_resync      state:D stack:0     pid:2365161 ppid:2      flags:0x00004000
Sep 10 10:40:38 rockylinux.com kernel: Call Trace:
Sep 10 10:40:38 rockylinux.com kernel: <TASK>
Sep 10 10:40:38 rockylinux.com kernel: __schedule+0x222/0x660
Sep 10 10:40:38 rockylinux.com kernel: schedule+0x5e/0xd0
Sep 10 10:40:38 rockylinux.com kernel: md_do_sync+0xcef/0x1030
Sep 10 10:40:38 rockylinux.com kernel: ? sched_clock_cpu+0xf/0x190
Sep 10 10:40:38 rockylinux.com kernel: ? membarrier_register_private_expedited+0xa0/0xa0
Sep 10 10:40:38 rockylinux.com kernel: md_thread+0xb0/0x160
Sep 10 10:40:38 rockylinux.com kernel: ? super_90_load.part.0+0x350/0x350
Sep 10 10:40:38 rockylinux.com kernel: kthread+0xe3/0x110
Sep 10 10:40:38 rockylinux.com kernel: ? kthread_complete_and_exit+0x20/0x20
Sep 10 10:40:38 rockylinux.com kernel: ret_from_fork+0x31/0x50
Sep 10 10:40:38 rockylinux.com kernel: ? kthread_complete_and_exit+0x20/0x20
Sep 10 10:40:38 rockylinux.com kernel: ret_from_fork_asm+0x11/0x20
Sep 10 10:40:38 rockylinux.com kernel: </TASK>
```

## Resolution

To conclude from the above findings, the `udev` rule for the RAID device may be triggering changes too early, leading to a race condition during the `md0_resync` task. Removing this rule fixes the issue.

One potential solution is to use the [`udevadm settle`](https://linux.die.net/man/8/udevadm) approach or introduce a delay to avoid the race condition from occurring.

Alternatively, monitoring the `md0` status before applying devices changes via a script, could help to ensure the `resync` completes without `udev` interference.

## Root Cause

This issue is related to common `udev` limitations. At the time of a `udev` query, some storage devices might not be accessible. There might be a delay between event generation and processing, especially with numerous devices involved, causing a lag between kernel detection and link availability. External programs invoked by `udev` rules, such as `blkid`, might briefly open the device, making it temporarily inaccessible for other tasks.

## References & related articles

[`udev` man page](https://linux.die.net/man/8/udev)  
[`mdadm` man page](https://linux.die.net/man/8/mdadm)  
[`udevadm` man page](https://linux.die.net/man/8/udevadm)


Udev Rules Causing RAID Device Creation to Hang


## Introduction

[Security Technical Implementation Guides (STIGs)](https://www.titania.com/resources/guides/disa-stig-compliance-explained) produced by the Defense Information Systems Agency (DISA) are heavily implemented in the Enterprise Linux ecosystem. The configurations provided allow for hardening and mitigation against cybersecurity threats.

When DISA STIG is implemented in Rocky Linux, there are occasions where you see messages in `dmesg` about reduced security. This article will explain one such message in `unhashed kernel memory addresses` and if any action is needed to be taken.

## Problem

When looking at `dmesg` you see a warning: *This system shows unhashed kernel memory addresses*.

## Symptoms

The following messages are shown:

```text
[    0.012710] **********************************************************
[    0.012710] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[    0.012711] **                                                      **
[    0.012711] ** This system shows unhashed kernel memory addresses   **
[    0.012711] ** via the console, logs, and other interfaces. This    **
[    0.012712] ** might reduce the security of your system.            **
[    0.012712] **                                                      **
[    0.012712] ** If you see this message and you are not debugging    **
[    0.012712] ** the kernel, report this immediately to your system   **
[    0.012713] ** administrator!                                       **
[    0.012713] **                                                      **
[    0.012713] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[    0.012713] **********************************************************
```

## Root Cause

This is caused by the `slub_debug=P` option being added to the kernel cmdline parameters during boot. This configuration is required by [DISA STIG in Enterprise Linux 8](https://stigviewer.com/stigs/red_hat_enterprise_linux_8/2020-11-25/finding/V-230279)  and [DISA STIG in Enterprise Linux 9](https://stigviewer.com/stigs/red_hat_enterprise_linux_9/2024-12-04/finding/V-257794) to help mitigate [use-after-free based attacks](https://cwe.mitre.org/data/definitions/416.html).

The message does not indicate any issues with the server’s operation and can be safely ignored.

## References & related articles (Optional)

[DISA STIG Explanation](https://www.titania.com/resources/guides/disa-stig-compliance-explained)
[DISA STIG Requirements for Enterprise Linux 8](https://stigviewer.com/stigs/red_hat_enterprise_linux_8/)  
[DISA STIG Requirements for Enterprise Linux 9](https://stigviewer.com/stigs/red_hat_enterprise_linux_9/)  
[DISA STIG SLUB Requirements for Enterprise Linux 8](https://stigviewer.com/stigs/red_hat_enterprise_linux_8/2020-11-25/finding/V-230279)  
[DISA STIG SLUB Requirements for Enterprise Linux 9](https://stigviewer.com/stigs/red_hat_enterprise_linux_9/2024-12-04/finding/V-257794)  
[DISA STIG for Rocky Linux 8.x](https://docs.rockylinux.org/books/disa_stig/disa_stig_part1/)


Unhashed Kernel Memory Error in DMESG


## Introduction

The `sosreport` tool is a key utility used by system administrators and support teams to collect comprehensive diagnostic information from a Rocky Linux system.

When the `sos` command is run, an `sosreport` is generated and gathers configuration details, system logs, and state information in a structured, compressed archive.

This allows a support team to troubleshoot a problem system with detailed logs, without needing the user to manually run multiple commands in order to achieve the same level of log collection.

An `sosreport` can potentially collect a lot of sensitive information about a system and thus knowing what information an `sosreport` gathers, can ease concerns of system administrators when running the `sos` tool.

## Problem

A Rocky Linux node runs into an issue and needs further in-depth analysis by a 3rd party.

## Resolution

To clarify what an `sosreport` collects, below is a detailed layout:

### Hardware information

* DMI tables (`dmidecode`)

* PCI device tree (`lspci`)

* Loaded kernel modules (`lsmod`)

* CPU/Memory statistics (`free`, `numa`, `/proc/cpuinfo`)

* BIOS and firmware configs from `/boot`

### Kernel and system state

* GRUB configuration (`/boot/efi`)

* Uptime, load, and system identity (`uname`, `hostname`, `uptime`)

* Kernel logs (`dmesg`)

* System journal snapshots (`journalctl_--no-pager`)

### Storage and filesystem

* Mounted filesystems (`mount`, `df`)

* LVM information (`vgdisplay`, `/etc/lvm`)

* Disk usage and I/O statistics (`/var/log/sar` if the `sysstat` package is installed and configured)

* `/etc/fstab` and `autofs` configs

### Network configuration

* IP address and routing tables (`ip addr`, `ip route`)

* Network interfaces, drivers, and statistics (`ethtool`)

* `firewalld` and `nftables/iptables` configs

* Snapshots of `/etc/NetworkManager`, `resolv.conf`, `nsswitch.conf`

### Services and daemons

* Enabled systemd units and logs (`systemctl`, journal slices)
  
* Service-specific config directories (`sssd`, `postfix`, `cups`)

* Init scripts and legacy SysV information

### Authentication and security

* PAM configurations (`/etc/pam.d`)

* `sudoers` file, `sshd_config`, SELinux policy state (`sestatus`)

* Audit rules and logs (`/etc/audit`)

### Logging configuration and data

* rsyslog and journald config (`rsyslog.conf`, `journald.conf`)

* Rotated logs config (`/etc/logrotate.d/`)

* Current and previous logs from `/var/log` (depending on if `logrotate` is setup)

### Package management

* Installed RPMs and installation times (`rpm -qa`)

* Repo configs (`/etc/yum.repos.d`, `/etc/dnf`)

### Process and system usage

* Running processes (`ps`, `pstree`)

* Open files (`lsof`)

* Resource limits and environment variables

### Configuration files

* Entire `/etc` tree including symlinked files

* Configuration fragments in `/etc/sysconfig`, `/etc/modprobe.d`, `/etc/security`, etc

### Sosreport plugins and logs

* `sos_commands` contains outputs of all CLI commands executed

* `sos_logs` for plugin-level logging

* `sos.conf`, `groups.d`, `presets.d` reflects plugins used

## Recommendations for environments containing sensitive data

For environments that handle sensitive data, it is recommended to obfuscate the data before the `sosreport` leaves the premises. Please see 
[Generating sosreports on Rocky Linux](https://kb.ciq.com/article/rocky-linux/rl-generate-an-sosreport) under `Obfuscating an sosreport` for more information on how to obfuscate sensitive data in an `sosreport`.

## Root cause

Concerns about the data that `sosreport` collects in environments where sensitive data needs to be handled with care.

## Notes

`sosreport` should be run as root to ensure complete data collection.

Custom plugins or group selections can limit what is captured.

Data collection of an `sosreport` may be expanded or reduced via `/etc/sos/sos.conf` or CLI flags such as `--only-plugins`.

## References & related articles

[sosreport Source Code](https://github.com/sosreport/sos)


What Data is Collected in an sosreport on Rocky Linux


## Introduction

When attempting to install Rocky Linux over a slow or unreliable network connection including through IPMI (Intelligent Platform Management Interface), you may encounter timeout issues that prevent the installer from loading properly.

## Problem

When the installer fails to load within the expected time frame, you may encounter the following error message:

```text
X did not start in the expected time, falling back to text mode.
```

This error occurs because the graphical installer (X server) is unable to start within the default timeout period. This is often caused by slow network speeds, high latency, or interruptions in the connection, which delay the loading of necessary resources. This issue can also occur when a server is configured to use 802.3ad (LACP) for link aggregation but is not configured to support this during the iPXE boot process. As a result, the network interfaces may fail to establish a proper connection, leading to network communication problems. As a result, the installer fails over to text mode, which may not be desired.

## Resolution

If switching to a faster network connection is not an option, there are two primary methods to resolve this issue:

1. **Increase the timeout period** to give the installer more time to load.
2. **Load the installation media into memory**, which bypasses the need to continuously fetch data over the network.

Both methods involve editing the kernel parameters during the installation process.

### Increase timeout

The default timeout for the installer is 60 seconds. You can increase this timeout by modifying the kernel parameters.

1. During the boot process, press `e` to edit the kernel boot options.
2. Locate the line that begins with `linux` or `linuxefi` and contains the kernel parameters.
3. Add the following parameter to the end of the line:

   ```text
   inst.xtimeout=<SECONDS>
   ```

   For example, to set the timeout to 5 minutes (300 seconds), you would add:

   ```text
   inst.xtimeout=300
   ```

4. Press `Ctrl-X` or `F10` to continue booting with the new parameter.

The installer will now wait up to the specified timeout period before falling back to text mode.

### Load into memory

If increasing the timeout does not resolve the issue or if the network connection is highly unstable, you can force the installer to load the entire installation media into memory. This eliminates the need to continuously fetch data over the network during the installation process.

1. As with the previous method, press `e` to edit the kernel boot options.
2. Add the following parameter to the end of the line:

   ```text
   rd.live.ram
   ```

   This option instructs the installer to copy the entire ISO or installation image into memory.

3. Continue with the installation.

Depending on the size of the installation media and the speed of the network, this process may take some time and you will not see any progress or status update.

## Notes

Loading the installation media into memory can require significant RAM. The amount of memory needed depends on the size of the ISO or image and the packages included. For example, a minimal Rocky Linux installation image may require only a few gigabytes of RAM, while a full *DVD* image with additional packages could require significantly more.

If you’re working in a slow or unreliable network environment and need to install or manage multiple servers, it may be worth setting up a [local mirror](https://kb.ciq.com/article/create-local-repo-using-reposync). A local mirror allows you to host the necessary installation files and package updates within your network, significantly reducing dependency on external connections.

## References & related articles

[How to Create a Local Mirror Using Reposync with CIQ Depot](https://kb.ciq.com/article/create-local-repo-using-reposync)  


X Timeout When Installing Rocky Linux


## Introduction

Multiple UNIX systems utilize the 32-bit `time_t` type, which is part of the Standard C Library (`libc`). Come January 2038, the `time_t` type will run out of bits and therefore be unable to show the current time, affecting countless applications.

This article will dive into what the Y2K38 bug means for Rocky Linux and also quell any fears that the bug will affect the popular operating system.

## Problem

On January 2038, the `time_t` type will not have enough bits available to store the current time, therefore causing multiple 32-bit reliant systems to fail.

An example is an `ext4` filesystem with an inode size of `128`. Once January 2038 hits, this then causes file timestamps to not be correctly recorded.

## Symptoms

32-bit clocks will start to overflow and come back with false values. Example real-world applications that would be impacted are financial forecasts and long-term mortgage interest calculations.

## Resolution

Thankfully, steps have already been taken to mitigate the Y2K38 issue. These include:

* [time_t improvements in the Kernel 5.6 onwards](https://lkml.org/lkml/2020/1/29/355), notably replacing `time_t` with safer alternatives.

**_⚠️ WARNING_** _Caveats do exist however, including ensuring that user space is compiled with the 64-bit version of `time_t` and other considerations._

* [XFS inode timestamps were increased in Kernel 5.10](https://lore.kernel.org/lkml/20201014205059.GD9837@magnolia/) to support dates up till the year 2486.

All current versions of Rocky Linux including 8, 9 and 10 include the above-mentioned patches in their kernels.

* For `ext4` users - if you're running with an inode size of `128` or less, you will need to move the data to a fresh `ext4` partition that has been configured with a higher inode size. It is possible to check the inode size, using the `tune2fs` tool like in the below example:

```bash
tune2fs -l /dev/sdb1 | grep "Inode size"
# In this case, the inode size is at 256, so this ext4 partition is not affected by the Y2K38 bug
Inode size:              256
```

## Summary

The Y2K38 bug does not affect the latest versions of Rocky Linux 8, 9 or 10 at this time.

## References & related articles

[Arnd Bergmann - [GIT PULL] y2038: core, driver and file system changes](https://lkml.org/lkml/2020/1/29/355)  
[Darrick J. Wong - [GIT PULL] xfs: new code for 5.10, part 1](https://lore.kernel.org/lkml/20201014205059.GD9837@magnolia/)  
[François Marier - Upgrading an ext4 filesystem for the year 2038](https://feeding.cloud.geek.nz/posts/upgrading-ext4-filesystem-for-y2k38/)  
[The Register - Linux 5.10 to make Year 2038 problem the Year 2486 problem](https://www.theregister.com/2020/10/19/linux_5_10_y2k38_fixes/)  
[The Register - Need 32-bit Linux to run past 2038?](https://www.theregister.com/2020/01/30/linux_5_6_2038/)


Y2K38 Bug and Rocky Linux


## Introduction

The [Year 2038 Problem (Y2K38)](https://lwn.net/Articles/776435/) is a known issue affecting systems that rely on a 32-bit `time_t` value. On January 19, 2038, this value will overflow, causing time calculations to break on affected systems. While this might seem like a distant issue, some systems already exhibit symptoms, particularly when incorrect timestamps are recorded due to system clock inconsistencies. If files have access or modification timestamps set in the future, beyond a system’s expected date range, licensing mechanisms and other time-sensitive applications may fail. This article provides a resolution for this issue by identifying and correcting future-dated files, ensuring proper time synchronization, and preventing recurrence.

## Problem

System files with modification, access, or creation timestamps set in the future, could cause issues. Such examples include licensing mechanisms to detect system clock discrepancy, leading to errors.

## Symptoms

Running `stat <file>` shows files with the `Access` timestamp set in the future.

```shell
stat /etc/fstab
  File: /etc/fstab
  Size: 2295            Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 67428642    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:etc_t:s0
Access: 2038-01-18 21:14:07.000000000 -0600 # <-- date in 2038
Modify: 2024-08-05 09:53:08.492859120 -0500
Change: 2024-08-05 09:53:08.493859164 -0500
Birth: 2024-08-05 09:53:08.492859120 -0500
```

Applications may fail to start due to a timestamp validation. We have seen problems while hosting a licensed application, observing the error `System clock has been set back`.

## Resolution

Follow these steps to correct the timestamps:

### 1. Identify Files with Future Dates

To locate files with timestamps set in the future, run the following command:

```shell
# Replace YYYY-MM-DD with a realistic cut-off date for your server
find / -newermt "YYYY-MM-DD" -ls
```

Ensure that you review the list before proceeding.

### 2. Update Timestamps on Affected Files

Once you identify the files, you can update their timestamps using the `touch` command to reset the `Access` and `Modify` times to the current dates.

```shell
find / -newermt "YYYY-MM-DD" -exec touch {} \;
```

Alternatively, to set a specific date (e.g., 2023-11-04), use:

```sh
find / -newermt "YYYY-MM-DD" -exec touch -t 202311040000 {} \;
```

#### Handling Symbolic Links

By default, the `touch` command may not update timestamps of symbolic links. To ensure timestamps of symlinks are also modified, use:

```sh
find / -type l -newermt "YYYY-MM-DD" -exec touch -h {} \;
```

According to the [`touch man page`](https://man7.org/linux/man-pages/man1/touch.1.html):

```code
`-h, --no-dereference` affects each symbolic link instead of any referenced file (useful only on systems that can change the timestamps of a symlink).
```

### 3. Verify Timestamps

After applying the changes, verify that timestamps have been corrected:

```shell
find / -ls
```

### 4. Restart the Affected Application

Once timestamps are corrected, restart the application:

```shell
sudo systemctl restart <your_application_service>
```

## Root Cause

This issue is often caused by an incorrect system clock setting or a failing CMOS battery, which can lead to timestamps being improperly set.

To prevent recurrence:

- Check and replace the **CMOS battery** if needed.
- Ensure the **system clock is correctly set**.

## References & Related Articles

- [Year 2038 Problem](https://en.wikipedia.org/wiki/Year_2038_problem)
- [Linux Touch Command Documentation](https://man7.org/linux/man-pages/man1/touch.1.html)


Year 2038 Issue in Rocky Linux Due to Power Outage


## Introduction

This article describes how to configure persistent NFS mounts on Warewulf-provisioned compute nodes using Warewulf overlays and systemd unit files. This method provides more flexibility compared to traditional `/etc/fstab` entries.
Warewulf overlays allow injecting custom files and configurations into nodes. We will create an overlay containing systemd mount units to manage NFS shares.

### Prerequisites

Decide if you want a single overlay for all NFS mounts or separate overlays for different mount points. Separate overlays offer more granular control, allowing different nodes to receive different combinations of mounts. This guide demonstrates using a single overlay for multiple mounts. You can repeat this for any other mountpoints you wish to cover.

Ensure that your path is properly [exported](https://docs.rockylinux.org/guides/file_sharing/nfsserver/) on your server.

### Create Systemd Unit Files

First, create systemd `.mount` unit files for each NFS share on your Warewulf control server. The unit file name must correspond to the mount path, replacing slashes `/` with hyphens `-`. For example, a mount point of `/data/common` requires a unit file named `data-common.mount`.

Save the following example files (adjusting NFS server IP, export paths, and mount options as needed). For simplicity, we save them in `/root` on the Warewulf server.

**/root/home.mount:**

```ini
[Unit]
Description=NFS Mount /export/home to /home
Requires=network-online.target
After=network-online.target

[Mount]
What=10.10.10.5:/export/home
Where=/home
Type=nfs
Options=defaults,_netdev

[Install]
WantedBy=multi-user.target
```

**/root/data-common.mount:**

```ini
[Unit]
Description=NFS Mount /export/common to /data/common
Requires=network-online.target
After=network-online.target

[Mount]
What=10.10.10.5:/export/common
Where=/data/common
Type=nfs
Options=defaults,_netdev

[Install]
WantedBy=multi-user.target
```

### Create Warewulf Overlay

Next, create a Warewulf overlay and import the systemd unit files into the correct location (`/etc/systemd/system/`).

```shell
wwctl overlay create nfs-mounts
wwctl overlay mkdir nfs-mounts /etc/systemd/system/
wwctl overlay import nfs-mounts /root/home.mount /etc/systemd/system/home.mount
wwctl overlay import nfs-mounts /root/data-common.mount /etc/systemd/system/data-common.mount
```

### Enable Mounts at Boot

To ensure systemd automatically mounts these shares at boot, enable the units by creating symbolic links within the overlay structure on the Warewulf server.

```shell
wwctl overlay mkdir nfs-mounts /etc/systemd/system/multi-user.target.wants/

ln -s /var/lib/warewulf/overlays/nfs-mounts/rootfs/etc/systemd/system/home.mount \
      /var/lib/warewulf/overlays/nfs-mounts/rootfs/etc/systemd/system/multi-user.target.wants/home.mount

ln -s /var/lib/warewulf/overlays/nfs-mounts/rootfs/etc/systemd/system/data-common.mount \
      /var/lib/warewulf/overlays/nfs-mounts/rootfs/etc/systemd/system/multi-user.target.wants/data-common.mount
```

### Assign Overlay to Nodes/Profiles

Assign the overlay to the desired nodes. Using profiles is recommended for managing groups of nodes. Create a new profile for the NFS mounts and assign the `nfs-mounts` overlay to its `RuntimeOverlay` list. Runtime overlays are applied during the node boot process and updated continually while the run node is running.

```shell
wwctl profile add nfs-profile --comment="Profile for NFS systemd mounts"

wwctl profile set nfs-profile --runtime-overlays="nfs-mounts"
```

Add this profile to your compute nodes:

```shell
wwctl node set compute01 --profile="<existing-profiles>,nfs-profile"
```

### Build Overlays

Build the overlays for all your nodes with the build command:

```shell
wwctl overlay build
```

### Reboot and Verify

Reboot the compute nodes that now have the `nfs-profile` assigned. If you set this as a Runtime Overlay, you should see the systemd units available within a few minutes. You will need to manually start these services if you chose not to reboot. Once logged in, verify the NFS shares are mounted correctly.

```shell
mount | grep nfs
df -hT | grep nfs
systemctl status home.mount
systemctl status data-common.mount
```

If this is not mounted, you can check the logs with

```shell
journalctl -u data-common.mount
```

## Notes

* Remember that systemd unit file names for mounts must reflect the mount path, replacing `/` with `-` (e.g., `/data/common` becomes `data-common.mount`). This is critical for systemd to mount the folders correctly.
* Ensure the NFS server IP address, exported paths, and mount options in the `.mount` files are correct for your environment.
* The `Requires=network-online.target` and `After=network-online.target` directives along with the `_netdev` mount option help ensure networking is available before attempting the NFS mount.

## References & related articles

[Warewulf Overlays Documentation](https://warewulf.org/docs/v4.6.x/overlays/overlays.html)  
[Warewulf Node Profiles Documentation](https://warewulf.org/docs/v4.6.x/nodes/profiles.html)  
[systemd.mount Documentation](https://www.freedesktop.org/software/systemd/man/systemd.mount.html)  
[Rocky Linux NFS Service Guide](https://docs.rockylinux.org/guides/file_sharing/nfsserver/)


Mounting NFS Shares on Warewulf Nodes Using Systemd Overlays


## Introduction

This article will cover assigning an IP address via DHCP using Warewulf, to a device that is not managed directly by Warewulf itself. This is useful in instances where you want to assign an address to a Network Attached Storage (NAS) device or similar non-compute node.

The implementation can be done by modifying the included DHCP template that comes with Warewulf.

## Resolution

First, backup the DHCP template (this is optional but recommended):

```bash
# For Warewulf 4.5.x
cp /var/lib/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww /var/lib/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.bak

# For Warewulf 4.6.x
cp /usr/share/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww /usr/share/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww.bak
```

There are two implementation methods available: adding the entries directly or creating a separate entry under `/etc/dhcp/hosts.conf` and including each host there. Both of these will be explained below.

### Option 1 - Add entries directly

You'll add each individual entry directly to the `dhcpd.conf` file. You will use the following entry as an example:

```bash
host MyDevice {
    hardware ethernet 00:11:22:33:44:55;
    fixed-address 10.0.0.5;
}
```

Edit the following template file `/var/lib/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww` for Warewulf 4.5.x using [vi](https://docs.rockylinux.org/books/admin_guide/05-vi/) or the text editor of your choice.

Alternatively for Warewulf 4.6.x, edit this template file instead: `/usr/share/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww`

 At the bottom of the `dhcpd.conf.ww` file, we are looking for the line `{{/* dhcp enabled */}}`.

 Move this under the `{{- end}}` line.

 In between the `{{/* dhcp enabled */}}` and `{{- end}}` lines, add your host entry, similar to the example below:

```bash
...
{{end -}}{{/* range NetDevs */}}
{{end -}}{{/* range AllNodes */}}
{{end -}}{{/* if static */}}
{{- else}}
{{abort}}
{{- end}}

host MyDevice {
    hardware ethernet 00:11:22:33:44:55;
    fixed-address 10.0.0.5;
}

{{/* dhcp enabled */}}
```

Add as many hosts as required.

Run the `wwctl configure dhcp` command to update the DHCP config.

Verify your hosts were added correctly using `cat /etc/dhcpd.conf`.

### Option 2 - Include a separate hosts file

Instead of adding each host to the template individually, you can also create a separate file and then include this in the main template.

Create a `hosts.conf` file under `/etc/dhcp/`.

Edit the `hosts.conf` file and add each individual host. Please see example host additions below:

```bash
host MyDevice {
    hardware ethernet 00:11:22:33:44:55;
    fixed-address 10.0.0.5;
}

host MyOtherDevice {
    hardware ethernet AA:BB:CC:DD:EE:FF;
    fixed-address 10.0.0.6;
}
```

Now edit the template.

Similar to "Option 1 - add entries directly", we are looking for the `{{/* dhcp enabled */}}` line in the file.

Move the `{{/* dhcp enabled */}}` line under the `{{- end}}` line.

In the middle of the `{{/* dhcp enabled */}}` and `{{- end}}` lines, add your host information.

Edit the `dhcpd.conf.ww` template file.

For Warewulf 4.5.x, this file is located under `/var/lib/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww`.

For Warewulf 4.6.x, this file is found under `/usr/share/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww`.
Exactly as before, you want to find the `{{/* dhcp enabled */}}` line.

Move this under the `{{- end}}` line.

Add your newly created `hosts.conf` via the `Include` line between the `{{- end}}` and `{{/* dhcp enabled */}}` lines. Below is an example:

```bash
...
{{end -}}{{/* range NetDevs */}}
{{end -}}{{/* range AllNodes */}}
{{end -}}{{/* if static */}}
{{- else}}
{{abort}}
{{- end}}

{{ Include "/etc/dhcp/hosts.conf" }}

{{/* dhcp enabled */}}
```

Again, run the `wwctl configure dhcp` command to update the DHCP config.

You can verify your hosts were added by checking the output of the `cat /etc/dhcpd.conf` command.

## Conclusion

You have successfully assigned an IP address to a device via DHCP using Warewulf, therefore opening a world of possibilities to assign IPs to any device on your network; not just compute nodes. This shows just how flexible Warewulf is as a DHCP server, as well as for cluster management.

## References & related articles

VI Text Editor: [Link](https://docs.rockylinux.org/books/admin_guide/05-vi/)  
Warewulf Configuration: [Link](https://warewulf.org/docs/main/contents/configuration.html#warewulf-conf)  


How to Add a Static DHCP Entry into Warewulf 4.5.x and 4.6.x


## Introduction

By default, Warewulf automatically fetches the latest kernel from the assigned container image and provides that kernel during PXE booting. Using the auto-detected kernel is recommended in the vast majority of cases. In addition, performing `dnf update` within the container should update to the latest kernel available.

## Problem

Some container images are created for specific Rocky Linux releases (e.g., Rocky Linux 8.8) and they would not have the right repositories to fetch updates, since they are built for that specific reason. Therefore the update process will not complete successfully.

## Symptoms

You may see the following when running `dnf update`:

```shell
# dnf update
Rocky Linux 8.8 - AppStream                                              4.6 MB/s |  12 MB     00:02
Rocky Linux 8.8 - BaseOS                                                 6.2 MB/s | 7.2 MB     00:01
Rocky Linux 8.8 - Extras                                                  24 kB/s |  14 kB     00:00
Dependencies resolved.
Nothing to do.
Complete!
```

## Resolution

There are few options available:

### 1 - Use a different image

Install a different image to make sure you can upgrade or downgrade to distinct versions. To change the container for your nodes without affecting operations, please follow [Best Practices for Applying OS Updates Across Node Images](https://kb.ciq.com/article/warewulf/ww-node-os-update).

### 2 - Use the `dnf upgrade` command

Use the `dnf upgrade` command instead and specify the release version you want to upgrade to:

```shell
$ dnf upgrade --releasever=8.10
Transaction Summary
=============================================================================================================================
Install   24 Packages
Upgrade  127 Packages

Total download size: 593 M
Is this ok [y/N]:
```

If you run into the following error while running the above command:

```shell
Errors during downloading metadata for repository 'appstream':
  - Status code: 404 for http://dl.rockylinux.org/vault/rocky/8.10/AppStream/x86_64/os/repodata/repomd.xml (IP: 151.101.54.132)
Error: Failed to download metadata for repo 'appstream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
```

Make sure to comment out the `_baseurl_` line in the `/etc/yum.repos.d` directory and uncomment the `_mirror_` line. You can use the command below:

```shell
sed -i -E 's/^(baseurl)/#\1/; s/^#(mirror)/\1/' /etc/yum.repos.d/*
```

Verify changes and try again.

### 3 - Change the repositories within the container image

You can manually set the `/etc/yum.repos.d/*` repositories to your preferred URLs for the version you want to upgrade to.

## References & related articles

[Warewulf Documentation - Images](https://warewulf.org/docs/v4.6.x/images/images.html)


Updating a Container's Kernel in Warewulf

## Introduction

This article addresses how to distribute files to configured and active compute nodes within a Warewulf managed cluster. It outlines three different methods: using Warewulf's native runtime overlay system, leveraging Ansible for configuration management, and using traditional `scp` or `pscp` commands.

## Problem

The user needs a method to copy files from the Warewulf server to a specific location on all or certain active compute nodes.

## Resolution

There are several ways to achieve this. Below are three common approaches:

### Option 1: Warewulf Runtime Overlays

Warewulf's overlay system is a powerful way to manage node configurations. Runtime overlays are particularly suited for distributing files or configurations that might need to be updated while nodes are running, as they are periodically pulled by the nodes from the Warewulf server. You can read more information about overlays in [Warewulf's documentation](https://warewulf.org/docs/v4.6.x/overlays/overlays.html). Here is a basic overview:

Follow these steps to distribute a file using a runtime overlay:

1. We start by creating an overlay for our files.

    ```bash
    wwctl overlay create my_file_distribution_overlay
    ```

2. Next, we will copy the local file into the overlay, specifying its destination path on the compute nodes. The `--parents` flag ensures that the directory structure is created within the overlay if it doesn't exist. Copy any other files you want into this as well.

    ```bash
    wwctl overlay import --parents my_file_distribution_overlay /path/to/your/local/file.txt /path/on/node/destination_file.txt
    ```

3. Now we can assign the overlay to a profile's runtime overlays. First, identify the existing runtime overlays for the target profile (e.g., `default`) to ensure you don't unintentionally remove them.

    ```bash
    wwctl profile list default -a | grep 'RuntimeOverlay'
    ```

    Let's say the output shows `default  RuntimeOverlay    hosts,ssh.authorized_keys`. You would then add your new overlay (`my_file_distribution_overlay`) to this list.

    ```bash
    wwctl profile set default -R hosts,ssh.authorized_keys,my_file_distribution_overlay
    ```

    **Note:** The `wwctl profile set -R` command replaces the entire list of runtime overlays for the specified profile with the new list you provide. Ensure you include all desired runtime overlays (both existing and new) in the command.

4. Finally, we can build the overlay to make it available to the nodes.

    ```bash
    wwctl overlay build
    ```

Once built and assigned, nodes configured with this profile will fetch and apply the runtime overlay, including your distributed file, during their next update cycle which usually runs every 5 minutes.

### Option 2: Using Ansible

If you are already familiar with and use Ansible in your environment, it can be a convenient way to distribute files.

1. Save the following content as a playbook YAML file (e.g., `distribute_file.yml`):

    ```yaml
    ---
    - name: Distribute a file to Warewulf compute nodes
      hosts: warewulf_compute_nodes # Ensure this group is defined in your Ansible inventory
      gather_facts: false
      tasks:
        - name: Copy the specified file to nodes
          ansible.builtin.copy:
            src: /path/to/your/local/file.txt
            dest: /path/on/node/destination_file.txt
            owner: root  # Optional: set file owner
            group: root  # Optional: set file group
            mode: '0644' # Optional: set file permissions
    ```

2. Run the playbook:

    ```bash
    ansible-playbook -i /path/to/your/inventory_file distribute_file.yml
    ```

This method is particularly useful if you need to perform other configuration tasks in conjunction with file distribution or if Ansible is already integrated into your workflow. For a complete automation solution, check out [Ascender](https://ciq.com/products/ascender/) Ascender is a fully open source alternative to Ansible Automation Platform or Ansible AWX that automates tasks like provisioning, configuration management, application deployment, and orchestration of your infrastructure and network processes.

### Option 3: Using SCP or PSCP (Parallel SCP)

#### SCP

The simplest method to copy a file over to your nodes would be to loop through a set of hosts and scp the file to your server. First, we need to create a hosts file (e.g., `hosts.txt`) containing the hostnames or IP addresses of your target compute nodes, one per line. By default, Warewulf will add the node names to your `/etc/hosts`, allowing you to reference them without needing to rely on the IP address. As an example:

 ```text
 # Example hosts.txt:
 node01
 node02
 node03
 ```

 Then, we can use a shell loop to `scp` the file:

 ```bash
 for host in $(cat hosts.txt); do
   scp "/path/to/your/local/file.txt" "root@${host}:/path/on/node/destination_file.txt"
 done
 ```

#### PSCP

If you have a lot of servers that you need a file copied to, you can take advantage of `pscp` from the `pssh` package. This will run the `scp` process in parallel. Using the same `hosts.txt` file:

 ```bash
 pscp.pssh -l "root" -h hosts.txt "/path/to/your/local/file.txt" "/path/on/node/destination_file.txt"
 ```

You can also use the `wwctl` command to pull nodes and filter based on certain criteria with `jq`. `jq` is a lightweight and flexible command-line tool for processing and querying JSON data. We will use `wwctl node list -j` and `jq` to dynamically target nodes. In this example, we will target all nodes in the `default` profile.

 ```bash
 pscp.pssh -l "root" -h <(wwctl node list -j | jq -r 'to_entries[] | select(.value.profiles | index("default")) | .key') "/path/to/your/local/file.txt" "/path/on/node/destination_file.txt"
 ```

## Notes

* To install `jq` in Rocky Linux, you can run `dnf install jq`, which will come from the `baseos` repository.

## References

[Warewulf Overlay Guide](https://warewulf.org/docs/v4.6.x/overlays/overlays.html)  
[Ansible Copy Documentation](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/copy_module.html)  
[Ascender Automation](https://ciq.com/products/ascender/)


How to Copy Files to Warewulf Nodes


## Introduction

Running CUDA in a Warewulf environment requires the proper setup of NVIDIA drivers and the CUDA toolkit. It is recommended to install the NVIDIA drivers directly within the Warewulf image while distributing the CUDA toolkit via NFS.

## Problem

One challenge when installing CUDA drivers directly into the Warewulf image is the increase in image size. Since the entire image is loaded into memory, this can cause booting issues especially in single-stage boot environments. This also reduces the memory available for applications. By moving the CUDA toolkit to external storage, these issues can be potentially avoided.

## Resolution

### Installing NVIDIA Drivers in the image

Follow this [guide](https://kb.ciq.com/article/warewulf/ww-installing-nvidia-drivers) to set up an image with Rocky Linux 9 and NVIDIA drivers.

### Setting up the CUDA NFS share

On your host node (or the server that will host the CUDA library), first download the appropriate CUDA version:

```shell
wget https://developer.download.nvidia.com/compute/cuda/12.8.1/local_installers/cuda_12.8.1_570.124.06_linux.run
chmod +x cuda_12.8.1_570.124.06_linux.run
```

Next, extract the toolkit:

```shell
./cuda_12.8.1_570.124.06_linux.run --toolkit --override --silent
```

By default, the files are extracted to `/usr/local/cuda-12.8` and a symbolic link is created at `/usr/local/cuda`, which is the location we will use. If you wish to extract the toolkit to a different location, append the option `--toolkitpath=<path>`.

Now, add the CUDA toolkit folder as an export with NFS. First, install the NFS utilities and start the NFS server:

```shell
sudo dnf -y install nfs-utils
sudo systemctl enable --now nfs-server
```

Then, add the export by appending the following line to your `/etc/exports` file. Replace `WAREWULF_IPRANGE` with the range or subnet of your Warewulf cluster network. Alternatively, you may use `*` to allow access from all IP addresses:

```shell
echo "/usr/local/cuda WAREWULF_IPRANGE(rw,sync,no_subtree_check)" | sudo tee -a /etc/exports
```

Finally, refresh the NFS exports:

```shell
sudo exportfs -ra
```

### Creating a systemd mount unit for CUDA

With the NFS export set up, you can create a `systemd` unit file to mount the CUDA toolkit on your nodes. Begin by creating a new overlay for this purpose:

```shell
sudo wwctl overlay create cuda-nfs
```

Next, create the `systemd` unit file. The naming convention requires that the filename contains the full mount path with slashes replaced by dashes. In this case, the file name should be `usr-local-cuda.mount` for mounting at `/usr/local/cuda`.

Open the file for editing:

```shell
sudo wwctl overlay edit cuda-nfs -p /etc/systemd/system/usr-local-cuda.mount
```

And add the following content, replacing `your_nfs_server` with your server's address:

```ini
[Unit]
Description=Mount NFS share for CUDA
After=network-online.target
Requires=network-online.target

[Mount]
# Change this to your server and export path
What=your_nfs_server:/usr/local/cuda
Where=/usr/local/cuda
Type=nfs
Options=defaults,noatime,nolock

[Install]
WantedBy=multi-user.target
```

Set appropriate permissions on the unit file:

```shell
sudo wwctl overlay chmod cuda-nfs /etc/systemd/system/usr-local-cuda.mount 0644
```

Next, create the symlink to ensure that the mount is activated at boot. In Warewulf `4.6.0` and above, we can create a symlink from within a template:

```shell
sudo wwctl overlay edit cuda-nfs -p /etc/systemd/system/multi-user.target.wants/usr-local-cuda.mount.ww
```

Add the following template line to the file:

```text
{{ softlink "/etc/systemd/system/usr-local-cuda.mount" }}
```

### Applying the overlay

Add the overlay as a system overlay to the default profile, by appending it to the existing overlay list:

```shell
sudo wwctl profile set default -O $(sudo wwctl profile list default -a | grep SystemOverlay | awk '{ print $3 }'),cuda-nfs
```

After updating the profile, rebuild the overlays:

```shell
sudo wwctl overlay build
```

Once the nodes have booted, the CUDA toolkit should now be mounted and available.

## Notes

- Ensure that your firewall settings allow NFS traffic.
- Adjust paths and options as needed based on your specific environment and security requirements.

## References & related articles

[Install NVIDIA Drivers in a Warewulf Container](https://kb.ciq.com/article/warewulf/ww-installing-nvidia-drivers)
[CUDA Downloads](https://developer.nvidia.com/cuda-downloads)  
[CUDA Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html)  
[Warewulf Overlay Documentation](https://warewulf.org/docs/v4.6.x/overlays/overlays.html)  
[Warewulf Profile Documentation](https://warewulf.org/docs/v4.6.x/nodes/profiles.html)  
[Warewulf Image Documentation](https://warewulf.org/docs/v4.6.x/images/images.html#importing-images)  
[systemd Mount Documentation](https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html)  
[Rocky Linux 9 Warewulf NVIDIA Containerfile](https://github.com/warewulf/warewulf-node-images/blob/main/examples/rockylinux-9-nvidia/Containerfile)  
[Rocky Linux NFS Guide](https://docs.rockylinux.org/guides/file_sharing/nfsserver/)  


Installing NVIDIA Drivers with CUDA in a Warewulf Image


## Introduction

When upgrading nodes in Warewulf using the `wwctl upgrade nodes` command, the `--add-defaults` flag plays a critical role in ensuring that default profiles and configurations are added. However, this can lead to issues in environments where no defaults were previously configured, potentially causing conflicts with existing setups. This article explains the behavior of the `--add-defaults` flag, its impact, and how to resolve related issues.

## Problem

Using the `--add-defaults` flag during `wwctl upgrade nodes` adds default profiles and configurations. In environments without pre-existing defaults, this can introduce settings that conflict with the current network or system configuration, leading to potential issues.

## Resolution

If you encounter issues after using the `--add-defaults` flag, follow these steps to resolve them:

### Step 1: Review added overlays

Inspect the overlays added by the upgrade process. These may include network-related overlays like:

- `ifcfg`
- `NetworkManager`
- `debian.interfaces`
- `wicked`

Run the following command to list the overlays:

```bash
wwctl overlay list
```

### Step 2: Customize the default profile

Modify the default profile to align with your environment by removing or adjusting problematic overlays.

### Step 3: Test and Validate

After making changes, test the configuration to ensure nodes operate as expected.

## Notes

The `--add-defaults` flag is mandatory for the wwctl upgrade nodes command, ensuring that essential configurations are added during the upgrade process. While the flag does not directly add network device configurations, conflicts may arise from how the new overlays interact with existing setups. Always back up your configuration files before performing upgrades.

## References

[Warewulf Documentation - Release Notes - version v4.6.0](https://warewulf.org/docs/v4.6.x/release/v4.6.0.html#the-default-profile)


Impact of defaults in Warewulf Upgrades


## Introduction

Warewulf configures an external DHCP service to direct cluster nodes how to boot from Warewulf. In its default configuration, Warewulf uses a DHCP pool to provision cluster nodes, after which they transition to the permanent IP address configured in the node registry.

---
**_⚠️ WARNING_** _The DHCP range must not overlap with the addresses that are assigned to nodes in the node registry._

---

To define the desired range of IP addresses to lease during the boot process, update `dhcp.range start` and `dhcp.range end`.

```yaml
dhcp:
  range start: 10.0.0.100
  range end: 10.0.0.110
```

---
**_NOTE:_** _The DHCP range is only used while a node is provisioning, and its size determines how many nodes can provision simultaneously. For example, a large cluster of thousands of nodes may be provisioned in batches of 32 nodes, so a DHCP pool of 32 addresses should be sufficient._

---

Warewulf may alternatively also be configured to provide static DHCP addresses based on individual node configurations. This requires additional administrative overhead, as the DHCP configuration must be updated when nodes are added to the cluster. It removes the need for a DHCP address pool to be separate from the assigned cluster node IP addresses.

```yaml
# warewulf.conf

dhcp:
  template: static
```

The Warewulf daemon should be restarted after the changes made to `warewulf.conf`, and the DHCP service must be reconfigured to generate the necessary static DHCP host entries.

``` bash
$ systemctl restart warewulfd

$ wwctl configure dhcp
Building overlay for wwctl1: host
Enabling and restarting the DHCP services

$ grep --after-context=5 '^host n1-default' /etc/dhcp/dhcpd.conf
host n1-default
{
    hardware ethernet 5a:00:05:23:eb:c6;
    fixed-address 10.0.0.111;
    option host-name "n1";
}
```

## Problem

Some users may try to change the `dhcp.template` option to `static` instead of the default config. As mentioned, it will require extra management to configure it the first time. This may cause the nodes to be unable to boot, as it may miss required information that has to be manually added in the `/etc/dhcpd.conf` file.

## Resolution

The DHCP configuration is only for booting the node. The best practice is to set the `dhcp.template:` to `default` and let the node boot with `autodiscovery`. Once it boots, it will get the IP address that was set on the node configuration. When this happens, Warewulf will save the MAC address in the `node.conf` file, and it will be easier to switch `dhcp.template:` to `static` in the future.

## References & related articles

[Warewulf v4.5.x Control Server Setup Documentation](https://warewulf.org/docs/v4.5.x/contents/setup.html)


DHCP Not Discovering Nodes if Set as Static


## Introduction

As specified in the [Warewulf Disk Management documentation](https://warewulf.org/docs/v4.5.x/contents/disks.html#rocky-linux), packages for [ignition](https://coreos.github.io/ignition/) are not available for the Rocky Linux 8.x series. They are available for the Rocky Linux 9.x series as part of the `Appstream` repository.

## Symptoms 

When you try to install `gdisk` or `ignition` on a Rocky Linux 8.x system, you will find it is not possible, due to the packages being unavailable in the repositories:

```bash
[root@Rocky-Linux-8-10-Test-Machine ~]# dnf install ignition
Last metadata expiration check: 0:04:50 ago on Fri 27 Dec 2024 05:46:44 AM UTC.
No match for argument: ignition
Error: Unable to find a match: ignition
```

## Resolution

### Prerequisites

You will require access to the `root` account or a user with escalated privileges.

### Build Ignition from source

When creating your container, add the following lines to your configuration:

```bash
&& (cd /tmp && git clone https://github.com/coreos/ignition.git) \
&& (cd /tmp/ignition && make all && make install) \
```

### Example container creation with Warewulf and ignition on Rocky Linux 8.10

Current disk structure of the nodes:

```bash
[root@Rocky-Linux-8-10-Warewulf-Controller-Node ~]# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sr0     11:0    1  1024M  0 rom  
vda    253:0    0   180G  0 disk 
├─vda1 253:1    0   260M  0 part /boot/efi
└─vda2 253:2    0 179.8G  0 part /
```

The Warewulf Rocky Linux 8.10 container was used for testing:

```bash
wwctl container import docker://ghcr.io/warewulf/warewulf-rockylinux:8 rockylinux-8 --build
```

**The following commands were all ran from the Controller Node:**

Created a 1GB `boot` partition for the Compute Node with `ext4` and wiped the filesystem at boot:

```bash
wwctl node set --diskname /dev/vda --diskwipe --partname boot --partsize=1024 --partnumber 1 --partcreate --fsname boot --fsformat ext4 --fspath /boot --fswipe
```

Assigned the rest of the disk space to the root filesystem, formatted with `btrfs`, and wiped the disk at boot for the Compute Node:

```bash
wwctl node set --diskname /dev/vda --partname root --partnumber 2 --partcreate --fsname root --fsformat btrfs --fspath / --fswipe
```

Output of how the Compute Node was configured:

```bash
[root@Rocky-Linux-8-10-Warewulf-Controller-Node ~]# wwctl node list -a warewulf-compute-node-1
  NODE                           FIELD                                                    PROFILE  VALUE                                                  
  warewulf-compute-node-1  Id                                                       --       warewulf-compute-node-1                          
  warewulf-compute-node-1  Comment                                                  default  This profile is automatically included for each node   
  warewulf-compute-node-1  ContainerName                                            default  rockylinux-8                                           
  warewulf-compute-node-1  Ipxe                                                     --       (default)                                              
  warewulf-compute-node-1  RuntimeOverlay                                           --       (generic)                                              
  warewulf-compute-node-1  SystemOverlay                                            --       (wwinit)                                               
  warewulf-compute-node-1  Root                                                     --       (initramfs)                                            
  warewulf-compute-node-1  Discoverable                                             --       false                                                  
  warewulf-compute-node-1  Init                                                     --       (/sbin/init)                                           
  warewulf-compute-node-1  Kernel.Args                                              --       (quiet crashkernel=no vga=791 net.naming-scheme=v238)  
  warewulf-compute-node-1  Profiles                                                 --       default                                                
  warewulf-compute-node-1  PrimaryNetDev                                            --       (default)                                              
  warewulf-compute-node-1  NetDevs[default].Type                                    --       (ethernet)                                             
  warewulf-compute-node-1  NetDevs[default].OnBoot                                  --       (true)                                                 
  warewulf-compute-node-1  NetDevs[default].Hwaddr                                  --       5a:00:05:3a:6f:5a                                      
  warewulf-compute-node-1  NetDevs[default].Ipaddr                                  --       10.25.96.4                                             
  warewulf-compute-node-1  NetDevs[default].Netmask                                 default  255.255.240.0                                          
  warewulf-compute-node-1  NetDevs[default].Gateway                                 default  10.25.96.3                                             
  warewulf-compute-node-1  NetDevs[default].Primary                                 --       (true)                                                 
  warewulf-compute-node-1  Disks[/dev/vda].WipeTable                                --       true                                                   
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[boot].Number                  --       1                                                      
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[boot].SizeMiB                 --       1024                                                   
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[boot].ShouldExist             --       true                                                   
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[root].Number                  --       3                                                      
  warewulf-compute-node-1  Disks[/dev/vda].Partitions[root].ShouldExist             --       true                                                   
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/boot].Format          --       ext4                                                   
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/boot].Path            --       /boot                                                  
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/boot].WipeFileSystem  --       true                                                   
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/root].Format          --       btrfs                                                  
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/root].Path            --       /                                                      
  warewulf-compute-node-1  FileSystems[/dev/disk/by-partlabel/root].WipeFileSystem  --       true
```

Ran `exec` to access the Rocky Linux 8.10 container:

```bash
wwctl container exec rockylinux-8 /bin/bash
```

Set up the environment, then cloned and built the `Ignition Partition Management Module` from inside the container:

```bash
dnf groupinstall -y "Development Tools"
dnf install -y gdisk git libblkid-devel make 
wget https://go.dev/dl/go1.23.4.linux-amd64.tar.gz
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.23.4.linux-amd64.tar.gz
```

* Added the following to the end of the ~/.profile file:

```bash
cat << "EOF" | tee ~/.profile
export PATH=$PATH:/usr/local/go/bin
EOF
```

* Added `~/.profile` to `$PATH`:

```bash
source ~/.profile
```

* Installed `ignition`:

```bash
cd /tmp && git clone https://github.com/coreos/ignition.git
cd /tmp/ignition && make all && make install
```

Ran `exit` to then leave and rebuild the container.

Deployed the container to the Compute Node by restarting the node.

## Notes

The above method is not officially supported by CIQ and is considered a workaround until the `gdisk` and `ignition` packages are available in the Rocky Linux 8.x repositories.

## References & related articles

Ignition Filesystem Documentation: https://coreos.github.io/ignition/operator-notes/#filesystem-reuse-semantics/

Warewulf Disk Management: https://warewulf.org/docs/v4.5.x/contents/disks.html#disk-management/


How to Setup Ignition with Warewulf on Rocky Linux 8.x


## Introduction

Kdump utilizes `kexec` to swiftly boot into a dump-capture kernel whenever a memory dump of the system kernel is required, such as during a system panic. This process preserves the system kernel's memory image across the reboot, making it accessible to the dump-capture kernel.

You will learn to how to enable `kdump` on one of your Compute Nodes in your Warewulf cluster. This is essential for when a kernel panic occurs and for performing an in-depth analysis at the kernel level.

## Prerequisites

_This guide assumes that a Warewulf server is successfuly installed and nodes are able to boot from a Rocky Linux image._

In many cases, the `kdump` service is installed and activated by default on new Linux installations, but some installation options do not have to install or enable `kdump` by default.

If you do not know whether `kdump` is installed on your system, you can "_shell into_" an image with `wwctl image shell` like the example below. For Warewulf 4.5.x versions, you can use `wwctl container shell`.

You can "_shell into_" the image with `wwctl image shell`. For 4.5.x versions you can use `wwctl container shell`.

```shell
[root@Warewulf ~]# wwctl image shell rockylinux-9
Image build will be skipped if the shell ends with a non-zero exit code.
```

From there you can check if the RPM has been installed by running `rpm -q kexec-tools`:

```shell
[warewulf:rockylinux-9] /# rpm -q kexec-tools
package kexec-tools is not installed
```

If you need to install `kexec-tools`, please run the command `dnf install kexec-tools`. This command secures installation of the userspace tools for `kexec`:

```shell
[warewulf:rockylinux-9] /# dnf install kexec-tools

Last metadata expiration check: 0:01:12 ago on Wed Mar  5 18:07:55 2025.
Dependencies resolved.
=============================================================================================================================================================================================
 Package                                        Architecture                           Version                                                  Repository                              Size
=============================================================================================================================================================================================
Installing:
 kexec-tools                                    x86_64                                 2.0.27-16.el9_5.1                                        baseos                                 477 k

...

Complete!
```

## Installation instructions

### Configure kdump

The Linux kernel accepts several boot-time arguments. For example, Warewulf currently specifies the following arguments by default:
`quiet crashkernel=no vga=791 net.naming-scheme=v238`.

You can specify a different set of kernel arguments for a node or profile using `--kernelargs`. This value is recorded on a node or profile as the `Kernel.Args` field.

**To set this at the profile level, run this command:**

```bash
wwctl profile set default --kernelargs 'quiet,crashkernel=512M,vga=791,net.naming-scheme=v238'
```

**Similarly, to set this at the node level, run this command:**

```bash
wwctl node set node01 --kernelargs 'quiet,crashkernel=512M,vga=791,net.naming-scheme=v238'
```

_To specify the memory reserved for the `kdump` kernel, set the `crashkernel=` option to the required memory value. For example, to reserve 128 MB of memory, you can use `crashkernel=128M`._

---
**_NOTE_:** _For newest Warewulf versions, `kernel args` is set as a list to be able to combine them between profiles and nodes, therefore, these values may get appended. You potentially have to negate default values by adding "`~`". An example is: `~crashkernel=no`._

---

### Changing the `excludes` file inside the container to allow copying the `/boot` directory to the nodes

Warewulf can exclude files from an image to prevent them from being delivered to the compute node. This is typically used to reduce the size of the image when some files are unnecessary. Patterns for excluded files are read from the file `/etc/warewulf/excludes` in the image itself. For example, the default Rocky Linux images exclude these paths:

```shell
/boot/
/usr/share/GeoIP
```

To remove `/boot/` from the `excludes` file, shell into the image:

```shell
[root@Warewulf ~]# wwctl image shell rockylinux-9
Image build will be skipped if the shell ends with a non-zero exit code.
[warewulf:rockylinux-9] /# sed -i '/\/boot\//d' /etc/warewulf/excludes
[warewulf:rockylinux-9] /# cat /etc/warewulf/excludes
/usr/share/GeoIP
```

Since `kdump` needs the `vmlinuz` file to boot the dump kernel, you need to add a symlink to `/usr/lib/modules/<kernel_version>/<vmlinuz file>`, as this file is not present on `/boot`:

```shell
[warewulf:rockylinux-9] /# KERNEL_VERSION=$(ls /usr/lib/modules | head -n 1)
ln -s /usr/lib/modules/$KERNEL_VERSION/vmlinuz-$KERNEL_VERSION /boot/vmlinuz-$KERNEL_VERSION
[warewulf:rockylinux-9] /# ls /boot/ | grep vmlinuz
vmlinuz-5.14.0-503.19.1.el9_5.x86_64
```

### Configuring the kdump target

When a kernel crash is captured, the core dump can be stored in various ways: as a file in a local file system, directly on a device, or sent over a network using the NFS (Network File System) or SSH (Secure Shell) protocols. Only one of the three options can be configured at a time. By default, the `vmcore` file is stored in the `/var/crash` directory of the local file system. To verify this, check the `/etc/kdump.conf` file's contents. In this example, NFS is used, since the Warewulf nodes will clear anything in memory after a reboot, and anything added to `/var/crash` will be removed.

An example from the `/etc/kdump.conf` file:

```shell
# nfs <nfs mount>
#           - Will mount nfs to <mnt>, and copy /proc/vmcore to
#             <mnt>/<path>/%HOST-%DATE/, supports DNS.
```

#### Configure `kdump` manually on the container

_Shell into_ the image and run the following commands:

```shell
[warewulf:rockylinux-9] /# grep -v ^# /etc/kdump.conf

auto_reset_crashkernel yes
path /var/crash
core_collector makedumpfile -l --message-level 7 -d 31

[warewulf:rockylinux-9] /# vi /etc/kdump.conf

# Edit the fields accordingly 
# If adding nfs please comment path

auto_reset_crashkernel yes
#path /var/crash
nfs nfs_server_ip:/kdump/path
core_collector makedumpfile -l --message-level 7 -d 31
```

#### Configuring kdump using overlays

To create the `kdump` overlay, please run:

```shell
$ wwctl overlay create kdump
```

Edit the `kdump.conf.ww` file (this command will create the overlay template):

```shell
$ wwctl overlay edit --parents kdump /etc/kdump.conf.ww
```

Add the following configuration after the `# This file is autogenerated by warewulf` line:

```bash
auto_reset_crashkernel yes
core_collector makedumpfile -l --message-level 7 -d 31

{{ if .Tags.kdump_nfs_location }}
nfs {{ .Tags.kdump_nfs_location }}
{{ else }}
path /var/crash
{{ end }}
```

Check that the new changes were applied successfully:

```shell
wwctl overlay show kdump /etc/kdump.conf.ww
```

Display the changes on a node to confirm it is working as expected:

```shell
$ wwctl overlay show -r node01 kdump /etc/kdump.conf

backupFile: true
writeFile: true
Filename: /etc/kdump.conf
# This is a Warewulf Template file.
#
# This file (suffix '.ww') will be automatically rewritten without the suffix
# when the overlay is rendered for the individual nodes. Here are some examples
# of macros and logic which can be used within this file:
#
# Node FQDN = node01
# Node Cluster = oso
# Network Config = <no value>, <no value>, etc.
#
# Go to the documentation pages for more information:
# https://warewulf.org/docs/main/contents/overlays.html
#
# Keep the following for better reference:
# ---
# This file is autogenerated by warewulf

auto_reset_crashkernel yes
core_collector makedumpfile -l --message-level 7 -d 31


path /var/crash
```

Change the tag to add the NFS path by using the `--tagadd` flag. In the following example, the path used was `172.16.131.5:/var/crash/`:

```shell
$ wwctl profile set default --tagadd kdump_nfs_location=172.16.131.5:/var/crash/
Are you sure you want to modify 1 profile(s): y
```

Render the changes on a node to confirm the changes:

```shell
$ wwctl overlay show -r node01 kdump /etc/kdump.conf
# Removing the extra lines to shorten the output
...

nfs 172.16.131.5:/var/crash/
```

Add the newly created overlay to the default profile, and boot the node:

```shell
wwctl profile set default -O $(wwctl profile list default -a | grep SystemOverlay | awk '{print $3,kdump}')
```

### Testing the kdump configuration

---
**_⚠️ WARNING_** _The commands below cause the kernel to crash. Use caution when following these steps, and by no means run them on a production system._

---

To test the configuration, `ssh` into the node, and make sure that the service is running:

```shell
$ systemctl status kdump.service 

#If the service is active, please run the following commands:

$ echo 1 > /proc/sys/kernel/sysrq
$ echo c > /proc/sysrq-trigger
```

The above commands force the Linux kernel to crash, and the `address-YYYY-MM-DD-HH:MM:SS/vmcore` file is copied to the location you have selected in the configuration (that is, to `/var/crash/` by default).

## References & related articles

[Kdump docs](https://www.kernel.org/doc/Documentation/kdump/kdump.txt)  
[kdump.conf man page](https://linux.die.net/man/5/kdump.conf)  
[Warewulf documentation](https://warewulf.org/docs/)  


How to Enable Kdump in Warewulf Nodes


## Introduction

In Warewulf 4.6.x you can mount NFS shares on provisioned nodes either by managing entries in `/etc/fstab` via overlays and profiles, or by dropping `systemd` mount units via overlays. The `systemd-unit` approach is now preferred because it avoids duplicate entries and leverages advanced templating.

## Problem

When you define the same NFS mount in multiple profiles the `fstab` overlay simply concatenates entries, leading to duplicate or unintended mounts on nodes. For example, if both `default` and `demo` specify a mount to `/mnt`, your rendered `/etc/fstab` may contain:

```text
demo-server:/export /mnt nfs defaults 0 0
default-server:/export  /mnt nfs defaults 0 0
```

The `fstab` overlay currently provides no built-in method for deduplication.

## Resolution

There are two recommended approaches:

### Use systemd mount units via overlays

Using `systemd` mount units provides greater overall flexibility. Not only can you create an individual overlay per mount and assign it to profiles or nodes, you can also apply Warewulf’s advanced templating and utilize features such as [tags](https://warewulf.org/docs/v4.6.x/nodes/nodes.html#tags) to dynamically generate your mounts. You can follow this guide on how to [add NFS shares via systemd](https://kb.ciq.com/article/warewulf/ww-add-nfs-share).

### Split fstab mounts into dedicated profiles

If you want to use multiple profiles, follow the steps below:

1. Backup your current configuration.
2. Create separate profiles for each NFS mount:

    ```bash
    wwctl profile create nfs-1
    wwctl profile create nfs-2
    ```

3. Populate `nfs-1` with your mounts:

    ```bash
    wwctl profile edit nfs-1
    ```

    ```yaml
    nfs-1:
        resources:
          fstab:
            - file: /mnt/folder1
              mntops: defaults
              spec: 10.0.0.15:/share/folder1
              vfstype: nfs4
            - file: /mnt/folder2
              mntops: defaults
              spec: 10.0.0.15:/share/folder2
              vfstype: nfs4
    ```

4. Repeat for `nfs-2` or any other profiles.
5. Apply the new profiles to your nodes appending them to your current profile:

    ```bash
    wwctl node set node01 --profile="default,nfs-1,nfs-2"
    ```

6. Verify profiles and rendered fstab:

    ```bash
    wwctl node list node01 --all
    wwctl overlay show --render node01 fstab /etc/fstab
    ```

7. Test on a non-production node before rolling out.

## Notes

- We do not typically recommend editing the node or profile YAML directly with `wwctl profile edit`, however, there is currently no other method for modifying resources.
- Always back up your Warewulf configuration before editing profiles, especially when editing the YAML directly.
- YAML spacing and indentation is very important and must be exact or Warewulf will reject the changes.
- Test changes on a non-production node to avoid disruption.

## References & related articles

[Add NFS Share via Warewulf Overlays](https://kb.ciq.com/article/warewulf/ww-add-nfs-share)  
[Warewulf Overlays](https://warewulf.org/docs/main/overlays/overlays.html)  
[Setting Warewulf Tags](https://warewulf.org/docs/v4.6.x/nodes/nodes.html#tags)


Managing fstab Overlays in Warewulf


## Introduction

This article addresses common inquiries related to High Availability (HA) configurations for Warewulf. While HA is not a standard or commonly documented use case for Warewulf, this article outlines key considerations, potential setups, and workarounds for environments that may require increased fault tolerance.

## Problem

Some users may inquire about implementing high availability for Warewulf in order to ensure uninterrupted node provisioning, particularly in large-scale compute environments. However, there is no official documentation or native HA support for Warewulf, which can make planning such deployments unclear.

## Resolution

Warewulf was not originally designed with high availability in mind. The main impact of an outage is limited to the provisioning of new or rebooted nodes, not the continued operation of already running services. This makes full HA less critical for many use cases, but potentially necessary in large or high-stakes environments.

Some users have successfully configured Warewulf HA using tools like [`Pacemaker` and `Corosync`](https://clusterlabs.org/projects/pacemaker/doc/2.1/Clusters_from_Scratch/singlehtml/), but there is no official documentation for this approach in the Warewulf project. Alternatively, you can maintain a cold or warm spare by keeping configuration files synced. This is often sufficient, since a Warewulf outage does not affect currently running nodes or services, only re-provisioning.

Although Warewulf does not provide built-in HA functionality, users may implement a basic level of high availability using the following approach:

1. Set up a primary Warewulf server to handle node provisioning and management.
2. Replicate key configuration files for example, sending `/etc/warewulf/nodes.conf` and `/var/lib/warewulf/` to one or more secondary servers. This can be achieved using tools like `rsync`, `cron`, or configuration management systems such as Ansible and Puppet.
3. Configure DHCP services carefully. One of the key challenges with Warewulf in a high availability (HA) setup is implementing a reliable HA DHCP solution.
   - Consider using [ISC DHCP failover](https://kb.isc.org/docs/aa-01356), which has basic support for HA configurations, though it is not widely implemented even in large environments.
   - Use non-overlapping DHCP pools across HA nodes to avoid IP conflicts.
   - Alternatively, implement static DHCP provisioning.
   - If your infrastructure includes a large number of nodes (10,000+), ensuring the availability of provisioning subsystems becomes more critical. In such scenarios, the network architecture typically involves multiple segments, also known as fat-tree architecture, with Warewulf and DHCP services distributed across different parts of the network.

## Notes

- Most HA considerations revolve around DHCP availability, which is more complex and less commonly implemented.
- Consider your risk profile: if node reboots are rare and the Warewulf server is stable, full HA may not be necessary.
- When designing an HA strategy, ensure thorough testing in a staging environment before deployment.

## References & related articles

- [ISC DHCP Failover Documentation](https://kb.isc.org/docs/aa-01356)  
- [Warewulf Documentation - Network Planning](https://warewulf.org/docs/main/getting-started/network.html)  
- [Pacemaker and Corosync Documentation](https://clusterlabs.org/projects/pacemaker/doc/2.1/Clusters_from_Scratch/singlehtml/)


Warewulf High Availability Considerations


## Introduction

There are multiple ways to create an image for Warewulf that includes the NVIDIA drivers. This article will explore how to work with a `Containerfile` or modify a regular image.

## Prerequisites

This guide assumes that a Warewulf server [has been installed](https://warewulf.org/docs/main/server/installation.html) and configured, and at least one node has been deployed.

## Instructions

### Create an image using a Containerfile

Warewulf supports creating images directly from containers, enabling you to build a custom image that includes the required NVIDIA drivers. The [Warewulf-images](https://github.com/warewulf/warewulf-node-images/blob/main/examples/rockylinux-9-nvidia/Containerfile) GitHub repository has an example `Containerfile` that we can work from. Start by installing `Podman` on the server you want to build the image on. To keep things simple, we will install this on our Warewulf host server to streamline the import process:

```shell
sudo dnf install -y podman
```

Next, we'll create a directory to add our `Containerfile` to:

```shell
mkdir rockylinux-9-nvidia
cd rockylinux-9-nvidia
```

Now create a new file called `Containerfile` and add the following to it:

```Dockerfile
sudo tee ./Containerfile <<EOF
FROM ghcr.io/warewulf/warewulf-rockylinux:9

RUN dnf -y install dnf-plugins-core epel-release kernel-headers \
    && dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/$(arch)/cuda-rhel9.repo \
    && dnf -y module install nvidia-driver:latest-dkms \
    && dnf -y install datacenter-gpu-manager \
    && dnf clean all \
    && for dir in /usr/src/kernels/*; do dkms autoinstall --kernelver $(basename $dir); done \
    && dkms status
EOF
```

Modify the `Containerfile` as needed to suit your environment. Next, build the image:

```shell
podman build -t rockylinux-9-nvidia:v1 .
```

You may replace `v1` with a tag or versioning system that better fits your personal preference or corporate policy if needed. If you see an error `cannot apply additional memory protection after relocation: Permission denied`, you may need to rerun with `SELinux` disabled or configured:

```shell
podman build --security-opt label=disable -t rockylinux-9-nvidia:v1 .
```

Once the build finishes, save this to a `tar` file and then import the image into Warewulf:

```shell
podman save -o rockylinux-9-nvidia-v1.tar localhost/rockylinux-9-nvidia:v1
wwctl image import file://rockylinux-9-nvidia-v1.tar rockylinux-9-nvidia-v1
```

Once the import finishes, you can then assign the image to a [profile or node](https://warewulf.org/docs/v4.6.x/nodes/profiles.html) as you would any other image. You can find an example of this in the following section within this guide. Another benefit of using `Containerfiles` is their ability to be integrated into a CI/CD pipeline to enable version control and automated building.

### Modifying a regular image

Warewulf includes the ability to shell into an image via a shell. This allows you to modify the image and quickly install components like NVIDIA drivers. You first need to start by either copying an existing image or importing a new image. You can use `wwctl image list` to view your current images if you would like to build off an existing one. You can, of course, edit any existing image; however, we always recommend working from a copy or creating a backup first.

In this example, you are going to start with a fresh copy of Rocky Linux 9. You can do so by importing an image from the Warewulf repository:

```shell
wwctl image import docker://ghcr.io/warewulf/warewulf-rockylinux:9 rockylinux-nvidia-9
```

In this example, the name of our image will be `rockylinux-nvidia-9`. You may change this to match your environment or to add additional information such as the date or version. Once the image is finished importing, you can shell into it and install the NVIDIA drivers:

```shell
wwctl image shell rockylinux-nvidia-9
```

You can find more detailed instructions for installing NVIDIA drivers [here](https://docs.nvidia.com/cuda/pdf/CUDA_Installation_Guide_Linux.pdf). The following are example steps for Rocky Linux 9:

```shell
# Install necessary packages and add the NVIDIA repository
dnf -y install dnf-plugins-core epel-release kernel-headers
dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/$(arch)/cuda-rhel9.repo
# Install the latest NVIDIA driver
dnf -y module install nvidia-driver:latest-dkms
```

We can verify this installed correctly by checking for the dynamically loaded kernel:

```shell
dkms status | grep nvidia
nvidia/580.95.05, 5.14.0-570.49.1.el9_6.x86_64, x86_64: installed
```

Once you have finished installing the driver and any other applications you need, type `exit` to leave the container. Warewulf should rebuild the container once you are finished, however you can run the following command manually to verify the container was built:

```shell
wwctl image build
```

Running this command will not return anything if the image has already been built/updated. If applicable, you can sync local users into the container. You can read more about the `syncuser` subcommand [here](https://warewulf.org/docs/v4.6.x/images/syncuser.html):

```shell
wwctl image syncuser --write rockylinux-nvidia-9
```

If you observe `WARN   : syncuser cannot determine what name should be assigned to id number` and `ERROR: error in synchronize` messages, add the users that are highlighted in the `WARN` message with `useradd <USERNAME>` and then run the `wwctl image syncuser` command again. You may need to add multiple users.

Once the `syncuser` command completes, you can assign this new image to a node. Assigning it directly to a node allows you to boot and test your newly created image before pushing it out to a profile. You can skip directly to assigning this to a profile if you desire:

```shell
wwctl node set node1 --image rockylinux-nvidia-9
```

Once you have tested your image, you can assign it to the necessary profile. In the example below, we are using the default profile:

```shell
wwctl profile set default --image rockylinux-nvidia-9
```

Finally, we rebuild the overlays:

```shell
wwctl overlay build
```

Upon reboot, your nodes will now boot with the newly created image! You can find more information about Warewulf images as well as more advanced configuration examples and instructions in the [Warewulf documentation](https://warewulf.org/docs/v4.6.x/images/images.html).

## References & related articles

[Warewulf Documentation](https://warewulf.org/docs/v4.6.x/images/images.html)  
[Warewulf Rocky Linux 9 NVIDIA Container Example](https://github.com/warewulf/warewulf-node-images/blob/main/examples/rockylinux-9-nvidia/Containerfile)  
[Hands on Warewulf: Solving Cluster Provisioning & Management](https://www.youtube.com/watch?v=7w8dAU9RSpE)  
[Warewulf: Deep Dive, Use Cases, and Examples](https://www.youtube.com/watch?v=EpVDeesAq4c)  


Install NVIDIA Drivers in a Warewulf Container


## Introduction
As mentioned [here](https://warewulf.org/docs/main/contents/ipmi.html):
It is possible to control the power or connect a console to your nodes being managed by Warewulf by connecting to the BMC through the use of IPMI

## Problem
The IPMI command works for all nodes after reboot, using their ipmitool command and node's IP address:
```
# ipmitool -I lanplus -H 10.0.0.11 -U root -P <PASSWORD> power status
Chassis Power is on
```

However, it does not work with wwctl power command:

```
wwctl power status cpu-20-10
ERROR  : cpu-20-10: Get Session Challenge command failed
Error: Unable to establish LAN session
Error: Unable to establish IPMI v1.5 / RMCP session
ERROR: exit status 1
```

## Resolution
You may need to configure IPMI to use **lanplus**. That could be achievable by setting `--ipmiinterface=lanplus`. Example:

```
wwctl profile set default \
> --netmask=255.255.255.0 --gateway=10.10.10.30 \
> --ipmiuser=root --ipmipass=[redacted] \
> --ipminetmask=255.255.255.0 --ipmiinterface=lanplus
```

## References & related articles
[Deploying a Dell PowerEdge and Cornelis Omni-Path Cluster with Warewulf](https://ciq.com/blog/deploying-a-dell-poweredge-and-cornelis-omni-path-cluster-with-warewulf/)

IPMI not working with wwctl power command for the LANPlus interface


## Introduction

`warewulfd` can generate massive log output when runtime overlays are transferred—especially across large clusters. This guide applies to Warewulf 4.6.x and shows how to limit that noise by adjusting the log level, helping you preserve disk space and focus on the logs that matter. This is the officially documented and recommended approach per the Warewulf project.

## Problem

By default, `warewulfd` logs each request and overlay file transfer at `INFO` and `SERV` levels. In clusters with many nodes, this results in gigabytes of log data being generated, making it hard to isolate real issues and consuming disk space rapidly.

## Symptoms

The system logs show repetitive entries such as:

```bash
wwctl[3151507]: request from hwaddr:xx:xx:xx:xx:xx:xx ipaddr:10.x.x.x | stage:runtime
wwctl[3151507]: SERV   : stage_file '/var/lib/warewulf/provision/overlays/node/__RUNTIME__.img'
wwctl[3151507]: send /var/lib/warewulf/provision/overlays/node/__RUNTIME__.img.gz -> node
```

These log messages appear once per provisioning request, per node.

## Resolution

To reduce the logging output, increase the `warewulfd` log level to `WARN` (this equates to a number of `30` in the configuration). This filters out routine `INFO` and `SERV` messages while still logging warnings and errors.

Set the new log level and restart the service:

```bash
echo WAREWULFD_LOGLEVEL=30 >> /etc/default/warewulfd
systemctl restart warewulfd.service
```

This will apply the new log level setting via the `EnvironmentFile` used by the `systemd` service.

## Root Cause

The default verbosity includes operational messages that are safe to suppress in most environments. At scale, these messages generate high log volumes unnecessarily.

## Notes

The following values are confirmed log level values for warewulf:

- INFO = 20
- SERV = 25
- WARN = 30

Restarting the service is required for the changes to take effect.

In Warewulf 4.6.0, [PR #1229](https://github.com/warewulf/warewulf/pull/1229) restructured the logging by removing the custom `SEND` and `RECV` log levels and moving transfer messages to the standard `INFO` level. The `WAREWULFD_LOGLEVEL` environment variable via `/etc/default/warewulfd` remains the supported mechanism for controlling log verbosity.

## References & related articles

[Warewulf Troubleshooting](https://warewulf.org/docs/v4.6.x/troubleshooting/troubleshooting.html)
[GitHub Issue #725 - warewulf server should be able to omit send/recv log messages](https://github.com/warewulf/warewulf/issues/725)
[GitHub PR #1229 - Simplify wwctl server command and logging](https://github.com/warewulf/warewulf/pull/1229)


How to Modify Warewulf Logging


## Introduction

This article outlines the best practices for applying OS updates across node images in a Warewulf environment. Following these practices helps ensure the stability and reliability of your system during updates.

## Problem

Administrators often face challenges when running OS updates against node images, including the risk of breaking the image or having difficulties in reverting changes. Ensuring a smooth update process is crucial for maintaining the operational integrity of the system.

## Symptoms

- Inconsistent node behavior post-update.
- Node failures or inability to boot.
- Difficulty in reverting to a previous stable state after updates.

## Resolution

To minimize risks and ensure a smooth update process, follow these best practices in order of preference:

### Best practices when updating images

When planning OS updates in a Warewulf environment, here are three approaches you may consider: the primary method builds a new image using an updated `Containerfile` which can be run via a CI/CD pipeline, the secondary method copies the current image to a new image for updating, and the final method directly updates the existing image, which is riskier as it will not retain the original, working image.

#### Primary method

1. **Update the Containerfile**: Apply updates to the `Containerfile` used to build the node image.
2. **Build a New Image**: Construct a new image from the updated `Containerfile`.
3. **Import into Warewulf**: Import the newly built image into Warewulf.
4. **Gradual Migration**: Gradually migrate nodes to the new image to monitor and ensure stability.

#### Secondary method

1. **Copy Current Image**: Duplicate the current node image and assign a new name to the copy.
2. **Apply Updates**: Implement the necessary updates on the copied image.
3. **Gradual Migration**: Gradually transition nodes to the updated image to ensure stability and functionality.

#### Tertiary method

1. **Direct Updates**: Apply updates directly to the current node image. This method is riskier and should only be used if the first two methods are not feasible.
2. **Backup**: Ensure there is a backup of the current image before proceeding with updates. This allows for a rollback if the update process fails or causes issues.

### Adding an image to nodes

When creating a new image, you have a couple options for applying the image within your environment, primarily at either the `profile level` or the `node level`. The configuration set directly on a node, overrides the configuration from a profile. For example, you can configure a node to use a different image than what's configured on the default profile.

#### Updating individual nodes

In this example, you will select a specific test node, **n8**, to upgrade to the newly built 8.10 Rocky Linux image. Please note that in Warewulf `4.5.8` and prior, the flag to add or edit a container image was `--container`. In `4.6.0` this was changed to `--image` to better reflect its purpose. However, the `--container` flag still works for compatibility.

```bash
$ wwctl node set n8 --image=rockylinux-8.10 

$ wwctl node list n8 --all
  NODE  FIELD                     PROFILE     VALUE
  n8    Id                        --          n8
  n8    Comment                   default     This profile is automatically included for each node
  n8    ImageName                 SUPERSEDED  rockylinux-8.10
  n8    Ipxe                      --          (default)
  n8    RuntimeOverlay            --          (generic)
  n8    SystemOverlay             --          (wwinit, [...])
  n8    Root                      --          (initramfs)
  n8    Init                      --          (/sbin/init)
  n8    Kernel.Args               --          (quiet crashkernel=no vga=791 net.naming-scheme=v238)
  n8    Profiles                  --          default,cluster1
  n8    PrimaryNetDev             --          (default)
  n8    NetDevs[default].Type     --          (ethernet)
  n8    NetDevs[default].OnBoot   --          (true)
  n8    NetDevs[default].Device   cluster1    eno1
  n8    NetDevs[default].Netmask  cluster1    255.255.255.0
  n8    NetDevs[default].Gateway  cluster1    10.0.0.3
  n8    NetDevs[default].Primary  --          (true)
```

The **ImageName** field for the `n8` node is now set to `rockylinux-8.10`. The **SUPERSEDED** value in the profile column indicates that a node value is replacing a profile value typically set from a profile. This allows you to test the new image on an individual server before rolling it out to a wider selection of nodes.

#### Updating a profile

You can also create a new profile. In this example, you will call it **dev** so it can be rolled out to multiple nodes at once. This is useful, particularly when testing, when you have multiple generations of hardware or configurations of hardware that need to be tested and validated to work with the new image. You will first start by creating the profile if it has not been created already.

```shell
$ wwctl profile add dev --comment="Development/testing environment"

$ wwctl profile list
  PROFILE NAME  COMMENT/DESCRIPTION
  dev           Development/testing environment
  default       This profile is automatically included for each node
```

Once this new `dev` profile is set, you can add information like you would on a node. In this case, apply a newer `rockylinux-8.10v2` image to this profile.

```shell
wwctl profile set dev --image=rockylinux-8.10v2
```

You can now add your node `n8` to this profile

```shell
wwctl node set n8 --profile=default,dev
```

While you can configure a node with one profile, one of the most powerful features of Warewulf is the ability to combine profiles. If two profiles set the same field, such as the `image` field, the right-most profile in the node’s list takes precedence. In the example above, the image from `dev` will take precedence over the image in `default`. Reminder: values set directly on nodes will take precedence over profile field values.
If you follow this guide, `n8` was set to use `rockylinux-8.10` as its image directly and as a result, this value overrides anything within the profiles. If you unset this value on the node, you will see `n8` use the `rockylinux-8.10v2` image:

```shell
wwctl node set n8 --image=UNSET
```

```shell
$ wwctl node list n8 -a
NODE  FIELD                        PROFILE  VALUE
----  -----                        -------  -----
n8    Profiles                     --       default,dev
n8    Comment                      dev      Development/testing environment
n8    ImageName                    dev      rockylinux-8.10v2
[...]
```

## Root Cause

The risk of breaking the image or encountering issues increases significantly when updates are applied directly to the current image. Using a `Containerfile` to build and update images or duplicating the current image before applying updates helps maintain stability and provides a clear rollback path.

## Notes

- Always ensure there is a backup of the current image before applying any updates.
- Gradual migration allows for monitoring and quick identification of issues, reducing the impact on the overall system.

## References & related articles

[Warewulf Profile Set Documentation](https://warewulf.org/docs/v4.6.x/reference/wwctl_profile_set.html)  
[Warewulf Node Set Documentation](https://warewulf.org/docs/v4.6.x/reference/wwctl_node_set.html)  
[Warewulf Profiles Documentation](https://warewulf.org/docs/v4.6.x/nodes/profiles.html)  
[Warewulf Images Documentation](https://warewulf.org/docs/v4.6.x/images/images.html)


Best Practices for Applying OS Updates Across Node Images


## Introduction

In this article, we will guide you through the process of adding your NAS system to your Warewulf cluster.

This involves configuring a static mapping on the DHCP configuration of your Warewulf server, to ensure that your NAS system is properly recognized and assigned a static IP address.

## Problem

A NAS is needed to be added to the Warewulf cluster, rather than a standard compute node.

## Symptoms

The NAS system is not being assigned an IP address from the Warewulf DHCP server.

## Resolution

The steps below are for Warewulf version 4.6.x. Please upgrade to the latest version of Warewulf first if possible. Full instructions are available via the [Upgrading Warewulf to Version 4.6.x](https://ciq.com/blog/upgrading-warewulf-to-version-4.6.x/) blog post.

- Check the DHCP logs located at `/var/log/messages` or `/var/log/dhcpd.log` to see if the NAS' MAC address is available there. Look for a request made to the DHCP server from the NAS to obtain an IP address. Sometimes the MAC address is not easily identifiable in the logs if there are many nodes in the cluster.

- You can manually add static IP mapping to the `dhcpd.conf.ww` DHCP configuration file.

- Firstly, make sure to (re)create your `dhcpd.conf.ww` by running:

```bash
wwctl configure dhcp
```

- The previous command (re)writes the DHCP configuration and enables and (re)starts the DHCP service.

- Backup the `dhcpd.conf.ww` configuration file:


```bash
cp /usr/share/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww /usr/share/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww.bak
```

- Edit the configuration file with your text editor of choice:

```bash
vim /usr/share/warewulf/overlays/host/rootfs/etc/dhcp/dhcpd.conf.ww
```

- Scroll to the bottom of the file and add a new line between `{{- end}}` and `{{/* dhcp enabled */}}`. The configuration will then look like this:

```bash
...
{{- end}}

{{/* dhcp enabled */}}
```

- In the gap you have created, add the following lines, supplementing the MAC address with the one from your NAS and the IP that you wish to assign to it:

```bash
...
host <IDENTIFIABLE_NAME_FOR_YOUR_NAS_HERE> {
    hardware ethernet <NAS_MAC_ADDRESS>;
    fixed-address <IP_TO_ASSIGN>;
}
```

- Here is an example configuration:

```bash
...
{{- end}}
host TEST {
    hardware ethernet A3:32:52:D1:2B:1C;
    fixed-address 192.168.1.254;
}
{{/* dhcp enabled */}}
```

- Apply the changes to the DHCP config:

```bash
wwctl configure dhcp
```

- The NAS should now be picked up by your Warewulf Controller Node upon next boot and assigned a static IP address.

## Root cause

A NAS system is sometimes not automatically recognized by the Warewulf DHCP server and needs to be manually assigned an IP address.

## Notes

Always backup the `dhcpd.conf.ww` configuration file before making any changes.

## References & related articles

[Upgrading Warewulf to Version 4.6.x](https://ciq.com/blog/upgrading-warewulf-to-version-4.6.x/)
[How to Add a Static DHCP Entry into Warewulf 4.5.x and 4.6.x](https://kb.ciq.com/article/warewulf/ww-add-static-dhcp)


Adding a NAS to Your Warewulf Cluster


## Introduction

PXE booting nodes using Warewulf 4.5.x and 4.6.x on Rocky Linux 8 can fail when `nftables` is active.

This article outlines a working approach to keep `nftables` enabled without blocking PXE boot traffic.

Users can work around the problem by disabling `nftables` entirely, which is not ideal for systems requiring persistent firewall rules.

## Problem

Enabling `nftables` on the Controller Node while using Warewulf on Rocky Linux blocks PXE boot traffic and thus new nodes cannot be attached to the cluster.

The challenge is creating a rule set that allows all necessary PXE and container traffic through without compromising security.

## Symptoms

Compute nodes fail to PXE boot when the Controller Node's `nftables` service is running.

Disabling `nftables` with `systemctl disable --now nftables` allows Compute Nodes to boot successfully.

## Resolution

### Prerequisites

- `root` or `sudo` privileges.

- Two or more physical nodes or VMs available for testing.

### nftables Configuration

- Ensure `nftables` is installed and enabled on the Controller Node:

```bash
dnf install -y nftables
```

- Flush any existing rules (make sure to backup any configurations first):

```bash
nft flush ruleset
```

- Create a new table called `filter` in the `inet` family:

```bash
nft add table inet filter
```

- Create a new chain named `input` inside the `filter` table (this will process incoming packets):

```bash
nft add chain inet filter input { type filter hook input priority filter ; }
```

- Add a rule to the chain to drop incoming traffic on `tcp` port `9873`:

```bash
nft add rule inet filter input tcp dport 9873 drop
```

- Permanently save the rule to the `/etc/nftables.conf` file:

```bash
nft list ruleset > /etc/nftables.conf
```

- Restart the `nftables` `systemd` service:

```bash
systemctl restart nftables
```

- Confirm PXE boot works with `nftables` enabled.

## Root cause

PXE and container provisioning in Warewulf require specific ports to be open.

A default or misconfigured `nftables` ruleset may reject packets that Warewulf requires, especially when broad reject rules are applied early in the chain.

TCP port 9873 is used by Warewulf for HTTP requests. Without this port open, additional nodes cannot be added to the cluster.

## References & related articles

[Warewulf Server Configuration Documentation](https://warewulf.org/docs/main/server/configuration.html)
[nftables Man Page](https://www.netfilter.org/projects/nftables/manpage.html)


Allow PXE Boot to Function on Warewulf with Rocky Linux 8 With nftables Enabled


## Introduction

As described in the [Warewulf Images documentation](https://warewulf.org/docs/v4.6.x/images/images.html):

*Since the inception of Warewulf over 20 years ago, Warewulf has used the model of the “Virtual Node File System” (VNFS) as a template image for the compute nodes. This is similar to a golden master image, except that the node file system exists within a directory on the Warewulf control node (e.g. a chroot()).*

*In hindsight, we’ve been using containers all along, but the buzzword just didn’t exist. Over the last 5-6 years, the enterprise has created a lot of tooling and standards around defining, building, distributing, securing, and managing containers, so Warewulf v4 now integrates directly within the container ecosystem to facilitate the process of VNFS image management.*

*If you are not currently leveraging the container ecosystem in any other way, you can still build your own chroot directories and use Warewulf as before.*

*It is important to understand that Warewulf is not running a container runtime on cluster nodes. While it is absolutely possible to run containers on cluster nodes, Warewulf is provisioning the container image to the bare metal and booting it. This container will be used as the base operating system and, by default, it will run entirely in memory. This means that when you reboot the node, the node retains no information about Warewulf or how it booted.*

## Problem

There could be cases where a node might fail to extract images for various reasons (BIOS settings, hardware-specific problems, etc.), and there is no straightforward method to force specific nodes to use uncompressed images while keeping others on compressed ones.

## Symptoms

When trying to provision the node with a compressed image, the server hangs during the PXE boot after downloading the container. After some minutes, the server might reboot with no warning message.

![Compressed Image](/images/articles/Compressedimage1.png)

## Resolution

There are two choices you can consider:

1. The simpler approach is to copy [default.ipxe](https://github.com/warewulf/warewulf/blob/main/etc/ipxe/default.ipxe) to `nocompress.ipxe`, update it appropriately, and then direct nodes to use it with the ipxe template node attribute.
2. Define a node tag (something like `ipxe_nocompress`) and look for it in `default.ipxe`. You could then control whether to use compressed images or not based on what’s defined.

## References & related articles

- [Default iPXE script](https://github.com/warewulf/warewulf/blob/main/etc/ipxe/default.ipxe)  
- [Warewulf Images documentation](https://warewulf.org/docs/v4.6.x/images/images.html)


PXE Boot with Compressed Images


## Introduction

When upgrading from Warewulf `4.5.8` to Warewulf `4.6.0` and above, there are significant changes to the configuration. We recommend reading the [release notes](https://warewulf.org/docs/v4.6.x/release/v4.6.0.html#upgrade) to better understand the differences. Failure to follow the instructions will result in an environment where your compute nodes may be unable to boot.

## Problem

When booting a compute node within a cluster, you encounter a Kernel Panic stating the system is *Unable to mount root fs on unknown-block(0,0)*.

![Kernel Panic Error: Unable to mount root fs on unknown-block(0,0)](/images/articles/unable-to-mount-root-fs-on-unknown-block/console_error.jpg)

## Root Cause

In Warewulf `4.5.8` and earlier, the necessary components to boot a compute node were present in one single overlay, `wwinit`. In Warewulf `4.6.0` and above, the components were separated into multiple overlays. The Warewulf config needs to be upgraded to reference these new overlays.

## Resolution

Ensure that you read the [release notes](https://warewulf.org/docs/v4.6.x/release/v4.6.0.html#upgrade) to determine what options apply for your environment. Before running anything, we recommend creating a backup of your warewulf host node. Typically you will need to run two commands:

```shell
wwctl upgrade config
wwctl upgrade nodes --add-defaults --replace-overlays
```

The first command will upgrade your `warewulf.conf` file and the second will upgrade your `nodes.conf` with the necessary changes to the overlays and the default profile. Both of these will attempt to update their respective configuration file in-place, retaining a copy of the previous version. You can read more about these flags below:

- `--add-defaults`: This adds the [new default settings](https://warewulf.org/docs/v4.6.x/release/v4.6.0.html#the-default-profile) to the default profile when those settings are absent. If you do not wish to add defaults, specify `--add-defaults=false`.
- `--replace-overlays`: This flag replaces any reference to the `generic` or `wwinit` overlays with the new set of overlays that replace their behavior. If you do not wish to replace overlays, specify `--replace-overlays=false`.

> **Warning:** The upgrade with the `--replace-overlays` flag should only be run once.

Once you finish the upgrade process, you can run `wwctl overlay build` and attempt to boot your compute nodes once again.

## Notes

You may also need to rebuild your image after the upgrade if you see a *No such file or directory* error message when attempting to boot. You can run `wwctl image build <image name>` to build the image if it is necessary.

## References & related articles

[Warewulf 4.6.0 Changelog](https://warewulf.org/docs/v4.6.x/release/v4.6.0.html)  
[Warewulf 4.6.1 Changelog](https://warewulf.org/docs/v4.6.x/release/v4.6.1.html)  
[Warewulf Overlay Documentation](https://warewulf.org/docs/v4.6.x/overlays/overlays.html)  


Unable to Mount Root FS After Warewulf Upgrade


## Introduction

This article explains how to resolve an SELinux relabeling issue that occurs when using Warewulf with dracut boot mode. The issue affects Warewulf version 4.5.8 and earlier, as version 4.6.0 includes the resolution.

## Problem

In Warewulf version 4.5.8 and earlier, dracut boot mode bypasses Warewulf's `wwinit` process, which includes automatic SELinux relabeling handled by the `90-SELinux` script. By default, dracut starts `/usr/sbin/init` directly, skipping Warewulf's initialization scripts that manage SELinux configuration.

## Resolution

### For Warewulf 4.5.8 and earlier versions

To resolve this issue, manually add `init=/init` to the kernel command line. This forces dracut to execute Warewulf's `/init` script instead of the default `/usr/sbin/init`. The `/init` script properly initializes the system and runs the SELinux relabeling process through the `90-SELinux` script.

#### Steps to configure

##### Method 1: Using wwctl node set

```bash
wwctl node set <nodename> --kernelargs="init=/init"
```

If you already have existing kernel arguments, you'll need to include them as well:

```bash
wwctl node set <nodename> --kernelargs="<existing_args>,init=/init"
```

##### Method 2: Using wwctl profile set

```bash
wwctl profile set <profilename> --kernelargs="init=/init"
```

### For Warewulf 4.6.0 and later versions

Starting with Warewulf version 4.6.0, the boot templates automatically include the `init=/init` parameter. This update ensures proper SELinux relabeling and eliminates the need for manual intervention. Developers implemented the fix in the GRUB template (`grub.cfg.ww`) and the iPXE template (`default.ipxe`).

## References

[Warewulf Documentation](https://warewulf.org/docs)  
[GitHub Issue #1573](https://github.com/hpcng/warewulf/issues/1573)


SELinux Relabeling Bypassed


## Introduction

The Warewulf project provides the `ssh.authorized_keys` overlay to distribute SSH keys to compute nodes. This allows users to securely authenticate to compute nodes without sharing or managing passwords. However, there are scenarios where root passwords may be required, such as for development, testing, or specific organizational needs.

## Problem

When using pre-built images, such as those provided by the [Warewulf community](https://github.com/orgs/warewulf/packages), the root password is not set by default. This can pose challenges when there is a need to log in directly to a console.

## Resolution

The simplest way to set a root password is to modify the image directly. Use the following command to open a prompt and set the root password for the image. Replace `rockylinux-9` with the name of your image:

```shell
wwctl image exec rockylinux-9 -- /usr/bin/passwd root
```

Once the image is rebuilt, reboot your compute nodes to apply the changes and verify that the root password has been set successfully.

## Notes

We recommend using a centralized user management solution such as LDAP or FreeIPA. You can follow [this guide](https://ciq.com/blog/how-to-install-the-freeipa-server-on-rocky-linux-9/) on how to install FreeIPA on Rocky Linux 9.

If you need to temporarily disable root passwords for console logins during testing or debugging, you can add the `PasswordlessRoot` tag to a node or profile:

```shell
wwctl node set n1 --tagadd PasswordlessRoot=True
```

You will need to ensure that the node also has the `syncuser` overlay applied either via the profile or at the individual node level:

```shell
wwctl node set n1 --runtime-overlays <existing overlays>,syncuser
```

**_⚠️ WARNING_** _For security reasons, this is not recommended for production environments._

## References

[Warewulf Community Images](https://github.com/orgs/warewulf/packages)  
[How to Install FreeIPA Server on Rocky Linux 9](https://ciq.com/blog/how-to-install-the-freeipa-server-on-rocky-linux-9/)  
[Warewulf syncuser Documentation](https://warewulf.org/docs/main/overlays/overlays.html#syncuser)  


Add a Root Password to Warewulf Compute Nodes


## Introduction

After pushing Warewulf image updates to compute nodes, SSH key-based authentication stops working, prompting for passwords instead. This article addresses the scenario where the Warewulf cluster SSH key is missing from the authorized_keys overlay after an image update.

## Problem

SSH access to compute nodes fails with password prompts instead of accepting SSH key authentication. Users who previously had passwordless SSH access to compute nodes can no longer connect using their SSH keys.

## Symptoms

When attempting to SSH to a compute node, the system prompts for a password instead of accepting the SSH key:

```bash
ssh root@node001
root@node001's password:
```

The `ssh.authorized_keys` overlay is present in the node configuration, but SSH key authentication still fails.

## Resolution

The issue occurs when the Warewulf cluster SSH key is missing from the head node's authorized_keys file, which is used to populate the compute node overlay. Follow these steps to restore SSH access:

### Verify the SSH Overlay Configuration

First, check if the ssh.authorized_keys overlay is configured for the affected node:

```bash
wwctl node list <node> -a | grep Overlay
```

You should see output similar to:

```bash
node001 RuntimeOverlay  default  hosts,ssh.authorized_keys,syncuser
```

The `ssh.authorized_keys` entry confirms the overlay is configured.

### Check the Current Overlay Content

Verify what SSH keys are currently in the overlay for the compute node:

```bash
wwctl overlay show -r <node> ssh.authorized_keys /root/.ssh/authorized_keys.ww
```

This displays the contents of the authorized_keys file that will be deployed to the compute node. If the cluster key is missing, you'll need to add it as shown below.

### Verify the Cluster Key Exists

Check that the cluster SSH key pair exists on the head node:

```bash
ls -l /root/.ssh/
```

Look for files named `cluster` and `cluster.pub`. These are generated for the cluster during Warewulf installation.

### Add the Cluster Key to Authorized Keys

Add the cluster public key to the head node's authorized_keys file:

```bash
cat ~/.ssh/cluster.pub >> ~/.ssh/authorized_keys
```

The `ssh.authorized_keys` overlay will, by default, use the server's `/root/.ssh/authorized_keys` file.

### Verify the Key Was Added

Confirm the cluster key is now present in the overlay:

```bash
wwctl overlay show -r <node> ssh.authorized_keys /root/.ssh/authorized_keys.ww
```

You should now see the cluster key in the output.

### Rebuild the Overlay

Rebuild the overlay to push the changes to the compute nodes:

```bash
wwctl overlay build
```

Since `ssh.authorized_keys` is typically a runtime overlay, changes should propagate to compute nodes within approximately 60 seconds.

### Test SSH Access

After waiting about a minute for the overlay to sync, test SSH access to the compute node:

```bash
ssh <node>
```

SSH key authentication should now work without prompting for a password.

## Notes

- The `ssh.authorized_keys` overlay is typically configured as a runtime overlay, meaning changes propagate automatically without requiring a node reboot.
- The cluster SSH key (`~/.ssh/cluster` and `~/.ssh/cluster.pub`) is used by Warewulf for internal cluster communication.
- Always verify the overlay configuration before and after making changes to ensure the proper keys are being deployed.

## References & Related Articles

[Warewulf Overlays Documentation](https://warewulf.org/docs/v4.6.x/overlays/overlays.html)  
[Warewulf Overlays Documentation: SSH Section](https://warewulf.org/docs/v4.6.x/overlays/overlays.html#ssh)


Unable to SSH to Compute Nodes After Image Update


## Introduction

Warewulf 4.6.0 introduced structural changes to how user account data is synchronized across compute nodes. This includes the introduction of a new `syncuser` overlay and changes in how `passwd` and `group` files are managed. These updates are designed to streamline user syncing, but require explicit configuration.

## Problem

In Warewulf 4.5.x, user syncing was often handled by embedding `/etc/passwd` and `/etc/group` content directly into overlays like `generic`. Running `wwctl overlay build` after adding users on the head node was typically enough to propagate changes.

With Warewulf 4.6.x:

- The new `syncuser` overlay is not automatically included in node profiles.
- `wwctl image build` no longer syncs users by default — it requires `--syncuser`.
- User and group files are sourced from sysconfdir (which defaults to `/etc`), rather than hardcoded paths.
- `wwctl image syncuser` writes changes by default but doesn't rebuild the image unless `--build` is used.

Without awareness of these changes, users may find that adding or modifying users on the head node no longer reflects on compute nodes.

## Symptoms

- New users are not present on compute nodes after image or overlay rebuild.
- Changes to UIDs/GIDs are ignored, even after re-syncing.
- The `syncuser` overlay appears empty or is missing if not manually configured.

## Resolution

To correctly sync users in Warewulf 4.6:

1. Attach the syncuser overlay to the node profile:

```bash
wwctl profile set <profile> --runtime-overlays <comma,separated,list,of,current,overlays>,syncuser
```

2. Ensure passwd/group are in `sysconfdir` (typically `/etc/`), or symlink them if necessary:

```bash
ln -sf /etc/passwd /path/to/sysconfdir/passwd
ln -sf /etc/group /path/to/sysconfdir/group
```

3. When importing images (previously containers), use the `--syncuser` flag:

```bash
wwctl image import --syncuser docker://<path_to_image>
```

4. Run image syncuser with `--build` if you want changes applied immediately:

```bash
wwctl image syncuser <image_name> --build
```

5. Rebuild overlays:

```bash
wwctl overlay build
```

## Root Cause

The Warewulf 4.6 update decoupled user syncing from overlays like `generic`, formalizing it under the new `syncuser` overlay. This adds clarity and modularity but requires updated workflows. Without explicitly enabling `syncuser`, user changes on the head node will not propagate.

## Notes

- [The changelog for `v4.6.0`](https://warewulf.org/docs/main/release/v4.6.0.html#syncuser) outlines the full `syncuser` redesign. Refer to it when upgrading or troubleshooting.
- Running `wwctl image syncuser` without `--build` updates the image’s user database but does not trigger a rebuild.

## References & related articles

[Warewulf 4.6.0 Release Notes – Syncuser Section](https://warewulf.org/docs/main/release/v4.6.0.html#syncuser)


Configuring User Sync with the Syncuser Overlay in Warewulf 4.6


## Introduction

This article addresses a common configuration issue in Warewulf where the server fails to start due to a missing systemd service name configuration.

## Problem

Warewulf displays a warning message indicating that it cannot start or restart the Warewulf server because no systemd name has been configured in the configuration file.

## Symptoms

When running `wwctl configure -a` users may encounter the following warning message:

```text
WARN   : Not (re)starting Warewulf server: no systemd name configured
```

## Resolution

To resolve this issue, you need to add the `systemd name` parameter to the Warewulf configuration file.

### Step 1: Edit the Warewulf configuration file

Open the Warewulf configuration file for editing:

```bash
sudo vi /etc/warewulf/warewulf.conf
```

### Step 2: Add the systemd name parameter

Locate the `warewulf:` configuration block and add the `systemd name: warewulfd` parameter. The configuration block should look like this:

```yaml
warewulf:
    port: 9873
    ...
    systemd name: warewulfd
```

### Step 3: Verify the configuration

After making the changes, you can run the following command to apply the configuration:

```bash
wwctl configure -a
```

You will see the following confirming the change has worked successfully:

```text
Enabling and restarting the Warewulf server
```

## Notes

- The `systemd name` parameter tells Warewulf which systemd service to manage when starting, stopping, or restarting the Warewulf server daemon. If you are using a different name, be sure to adjust this accordingly.
- Make sure to maintain proper YAML indentation when editing the configuration file, as incorrect indentation can cause parsing errors.

## References & related articles

[Warewulf Documentation - Server Configuration](https://warewulf.org/docs/v4.6.x/server/configuration.html)  
[Warewulf Documentation - wwctl configure command](https://warewulf.org/docs/v4.6.x/reference/wwctl_configure.html)


Warewulf Reports No Systemd Name Configured


## Introduction

A `tmpfs` filesystem is a temporary, memory-based filesystem that resides in virtual memory, providing extremely fast file access. It dynamically allocates RAM and can use `swap` space when needed, consuming only as much memory as required for its contents. However, its contents are volatile, meaning all data is lost when the filesystem is unmounted. See [tmpfs(5)](https://man7.org/linux/man-pages/man5/tmpfs.5.html) for more details.

## Problem

The user doesn't want `tmpfs` to use `swap` space. For example, swap space set aside for slurm jobs.

## Resolution

As mentioned in the [warewulf documentation](https://warewulf.org/docs/main/contents/boot-management.html), the `wwinit` module provisions to `tmpfs`. By default, `tmpfs` is permitted to use up to 50% of physical memory. This size limit may be adjusted using the kernel argument `wwinit.tmpfs.size`. (This parameter is passed to the size option during `tmpfs` mount.)

`wwinit.tmpfs.size` is only used by [dracut](https://man7.org/linux/man-pages/man8/dracut.8.html). To boot with dracut please follow [this guide.](https://warewulf.org/docs/main/contents/boot-management.html#booting-with-dracut).

If you aren’t using dracut this parameter won’t adjust the `tmpfs` limit, and is not currently configurable, aside from manually editing the [`init` file](https://github.com/warewulf/warewulf/blob/main/overlays/wwinit/rootfs/init#L32) in the overlay, or maintaining a copy of it locally.

`tmpfs` blocks may be swapped out, when there is a shortage of memory. `tmpfs` has a mount option to disable its use of `swap`:

``` bash
noswap - # Disables swap. Remounts must respect the original settings. By default swap is enabled.
```

This can also be managed in the [init file](https://github.com/warewulf/warewulf/blob/main/overlays/wwinit/rootfs/init#L32)

## Notes

Kernel args can still be set with dracut boot. However, `--kerneloverride` as currently implemented would require the ability to push the matching kernel modules into the dracut image, which is not currently supported.

## References & related articles

[Warewulf - Booting with Dracut](https://warewulf.org/docs/main/contents/boot-management.html#booting-with-dracut)  
[Man tmpfs page](https://man7.org/linux/man-pages/man5/tmpfs.5.html)  
[Man dracut page](https://man7.org/linux/man-pages/man8/dracut.8.html)


How to Limit Tmpfs for Taking Swap Space


## Introduction

As explained in the [Warewulf Provisioning Guide](https://warewulf.org/docs/main/contents/provisioning.html), Warewulf provisions nodes configured to PXE boot.  
If a local disk is present and bootable, change the boot order in the node’s BIOS—this step precedes provisioning.

When a node boots via PXE, its firmware initializes hardware and requests a `BOOTP/DHCP` address from the network. The DHCP server (either the Warewulf controller or an external server) responds with a **next-server** directive and a filename. PXE then downloads that file via TFTP and executes a bootloader (iPXE or GRUB).

The bootloader retrieves the kernel, image, and overlays from the Warewulf HTTP server.  
In single-stage provisioning, the image and overlays combine immediately as the kernel’s initial root filesystem.  
In two-stage provisioning, the bootloader loads a minimal initial `rootfs` (via dracut), and the kernel later retrieves and deploys the full image and overlays.

The `wwclient` daemon runs on the node after boot to refresh runtime overlays as needed.

## Problem

Nodes fail to boot after being set to boot from PXE.

## Resolution

Node boot failures can occur for several reasons. Follow these steps to isolate the issue.

### Verify Services

Ensure that you enable and start all necessary Warewulf and system services:

```bash
systemctl status httpd dhcpd tftp-server.socket warewulfd
```

If any of these services are **inactive** or **failed**, start and enable them:

```bash
systemctl enable --now httpd dhcpd tftp-server.socket warewulfd
```

Confirm that `warewulfd` is running correctly:

```bash
systemctl status warewulfd
```

A healthy service should display as **active (running)** and include logs like:

```bash
[Thu May 29 18:26:46 UTC 2025] send /etc/warewulf/ipxe/default.ipxe -> n1
[Thu May 29 18:26:46 UTC 2025] request from hwaddr:5a:00:05:78:dc:f8 ipaddr:10.0.0.103:8209 | stage:kernel
[Thu May 29 18:26:46 UTC 2025] SERV   : stage_file '/var/lib/warewulf/chroots/rockylinux-9.4/rootfs/lib/modules/.../vmlinuz'
```

If `warewulfd` is active but nodes still fail to retrieve boot files, proceed to **firewall and network checks**.

### Check firewall and network access

A misconfigured or active firewall is a common cause of PXE and iPXE boot issues.  
To quickly rule out the firewall as the problem, temporarily stop it:

```bash
systemctl stop firewalld
```

If you prefer to keep the firewall enabled, ensure Warewulf-related services and interfaces are correctly allowed:

```bash
firewall-cmd --permanent --zone=internal --add-interface=<your_interface>
firewall-cmd --permanent --zone=internal --add-service=warewulf
firewall-cmd --permanent --zone=internal --add-service=http
firewall-cmd --permanent --zone=internal --add-service=tftp
firewall-cmd --reload
```

Then verify:

```bash
firewall-cmd --list-all --zone=internal
```

Your configuration should include at least the following services:

```bash
services: http tftp warewulf
```

**_💡 TIP_** _Most PXE boot issues trace back to blocked TFTP or HTTP traffic, even when `warewulfd` appears healthy._

### Test TFTP server

Check that the TFTP service can serve files correctly:

```bash
tftp <tftpserverIPADDR> -c get /var/lib/warewulf/bin-x86_64-efi-snponly.efi
```

**_💡 TIP_** _Run this test from a node in the same subnet as the Warewulf headnode to verify network and TFTP service status._

### Check SELinux

Warewulf supports provisioning nodes with SELinux enabled, but the image must reside on a filesystem that supports labeling.

If you enable SELinux on a filesystem without labeling support, such as `ramfs`, the following warning may appear:

```bash
Skipping SELinux configuration: rootfs does not support SELinux contexts
```

To turn off SELinux during provisioning:

```bash
wwctl node set --kernelargs="selinux=0"
```

To keep SELinux enabled, set the root filesystem to `tmpfs`:

```bash
wwctl node set --root=tmpfs n1
```

**_⚠️ WARNING_** _In Warewulf v4.6.0 and v4.6.1, a bug in `wwinit` causes SELinux to fail due to improper labeling of the `/run/` filesystem. You can backport the fix manually if needed._

### iPXE troubleshooting

If you’re using iPXE (the default bootloader), access its command prompt by pressing `Ctrl + B` during boot.  
From the prompt, run commands similar to those in `default.ipxe` to troubleshoot boot issues.

For example:

```bash
set uri http://10.0.0.1:9873/provision/00:00:00:00:00:00
kernel --name kernel ${uri}?stage=kernel
imgextract --name image ${uri}?stage=image&compress=gz
imgextract --name system ${uri}?stage=system&compress=gz
imgextract --name runtime ${uri}?stage=runtime&compress=gz
boot kernel initrd=image initrd=system initrd=runtime
```

- `uri` defines the Warewulf server and identifies the node by MAC address.  
- `kernel` fetches the kernel image.  
- `imgextract` downloads and decompresses the image and overlays that form the node’s root filesystem.  
- `boot` loads the kernel and ramdisks to start the system.

**Optional:**

To boot into a pre-init shell for debugging, replace the last line with:

```bash
boot kernel initrd=image initrd=system initrd=runtime rdinit=/bin/sh
```

Use `rdinit` (not `init`) because Warewulf boots nodes using an initial ramdisk.

## References and related articles

[Warewulf iPXE Troubleshooting](https://warewulf.org/docs/main/troubleshooting/troubleshooting.html#ipxe)  
[Node Provisioning Guide](https://warewulf.org/docs/main/contents/provisioning.html)  
[Security and SELinux Configuration](https://warewulf.org/docs/main/server/security.html)


Troubleshooting PXE Booting and Node Provisioning


## Introduction

This article outlines the process of enabling two-stage booting using Dracut on an existing Warewulf image. Two-stage booting can be beneficial in environments where traditional single-stage booting with large image sizes causes issues due to limitations in the node's firmware or bootloader.

## Problem

Traditional single-stage booting in Warewulf directly loads the entire image as an `initramfs` using bootloaders like iPXE or GRUB. While effective for smaller images, this approach can lead to memory management issues and boot failures when dealing with large container images.

## Symptoms

You might encounter the following symptoms when trying to boot nodes with large images using single-stage booting:

* Boot failures or hangs during the initramfs loading phase.
  * You may also see a kernel panic with an error similar to `not syncing: UFS: Unable to mount root fs on unknown-block(0,0)`
* Error messages related to memory allocation or insufficient memory.
* Inconsistent boot behavior across different hardware.

## Resolution

Enable two-stage booting with `dracut` on an existing Warewulf image, by following the steps below:

### Prerequisites

A functional Warewulf server with an existing Rocky Linux image. If you do not already have an image available, please consult the official Warewulf [Image Management](https://warewulf.org/docs/v4.6.x/images/images.html) documentation for guidance on creating or importing an image.

Before proceeding, we strongly recommend backing up your current image or creating a copy to use as a development version. This ensures you can easily revert to a known good state if needed.

### Steps

#### Install the `warewulf-dracut` package in the image

This command uses `wwctl image exec` to run a command inside the specified image. In our example, `rockylinux-9`. The `--build=false` flag prevents the image from being rebuilt immediately after the command executes. Install the `warewulf-dracut` package directly from the [Warewulf GitHub releases](https://github.com/warewulf/warewulf/releases). Replace the image name with one that matches your environment, and ensure you are using the correct version of the `warewulf-dracut` package. Update the URL below to point to the version you intend to use:

```shell
wwctl image exec --build=false rockylinux-9 -- /usr/bin/dnf --assumeyes install https://github.com/warewulf/warewulf/releases/download/v4.6.1/warewulf-dracut-4.6.1-1.el9.noarch.rpm
```

#### Generate the dracut `initramfs` inside the image

This command again uses `wwctl image exec` to run `dracut` within the image:

```shell
wwctl image exec --build=false rockylinux-9 -- /usr/bin/dracut --force --no-hostonly --add wwinit --regenerate-all
```

#### Build the Warewulf image

Build the image separately, as the previous step may take some time to complete:

```shell
wwctl image build rockylinux-9
```

#### Booting via dracut

After completing these steps, your Warewulf image supports two-stage [booting with dracut](https://warewulf.org/docs/v4.6.x/images/images.html#booting-with-dracut).
You need to configure the nodes that require two-stage booting to use the `dracut` iPXE or GRUB menu entry.

For iPXE:

```shell
wwctl node set <node name> --tagadd IPXEMenuEntry=dracut
```

For GRUB:

```shell
wwctl node set <node name> --tagadd GrubMenuEntry=dracut
```

> **Note**: Warewulf version 4.6.0 removed the dedicated `dracut.ipxe` template that was first introduced in 4.5.3. This feature is now in the main `default.ipxe` template.

You can also set this tag within a [profile](https://warewulf.org/docs/v4.6.x/nodes/profiles.html).

## Notes

* Ensure that the version of `warewulf-dracut` installed in the image is the same as your Warewulf server version.
* In some cases, such as with Rocky Linux 8, you might need to remove the `/etc/machine-id` file within the image for Dracut to generate the initramfs correctly.
  * You can do this by running `wwctl image exec <image name> -- /usr/bin/rm -f /etc/machine-id` before running the `dracut` command. Add `--build=false` if you plan on building the image separately.
* The Dracut-based two-stage provisioning is considered a preview feature and may undergo changes in future Warewulf releases.

## References & related articles

[Warewulf Documentation - Image Management](https://warewulf.org/docs/v4.6.x/images/images.html)  
[Warewulf Documentation - Booting with Dracut](https://warewulf.org/docs/v4.6.x/images/images.html#booting-with-dracut)  
[Current Warewulf Releases](https://github.com/warewulf/warewulf/releases)  
[Troubleshooting PXE Booting](https://kb.ciq.com/article/warewulf/ww-troubleshoot-pxe-booting)  
[Best Practices for Applying OS Updates Across Node Images](https://kb.ciq.com/article/warewulf/ww-node-os-update)


Enabling Two-Stage Booting with Dracut on an Existing Warewulf Image


## Introduction

Warewulf is a powerful cluster provisioning system that has undergone significant enhancements in its 4.6.x release. This guide provides a step-by-step walkthrough for upgrading your existing Warewulf 4.5.x to version 4.6.x, ensuring a smooth transition while leveraging the latest features and improvements.

## Prerequisites

Before initiating the upgrade process, ensure the following:

- You are currently running Warewulf version 4.5.x.

- A complete backup of your Warewulf configuration files (warewulf.conf, nodes.conf, and overlays) is available.

- Familiarity with command-line operations and basic system administration tasks.

## Instructions

### 1. Review the Release Notes

Familiarize yourself with the changes introduced in Warewulf 4.6.x by reviewing the [official release notes](https://warewulf.org/docs/main/release/v4.6.0.html). Key updates include:

- Introduction of the wwctl upgrade command for streamlined [upgrades](https://warewulf.org/docs/v4.6.x/release/v4.6.0.html#upgrade).

- Transition from "containers" to "[images](https://warewulf.org/docs/v4.6.x/release/v4.6.0.html#images)" terminology.

- Major changes to the [syncuser](https://warewulf.org/docs/v4.6.x/release/v4.6.0.html#syncuser) option and the addition of the syncuser overlay.

- Removal of the `wwctl kernel` command and [kernel](https://warewulf.org/docs/v4.6.x/release/v4.6.0.html#kernels) overrides.

- Enhanced [overlay](https://warewulf.org/docs/v4.6.x/release/v4.6.0.html#overlays) management.

- The default settings moved to the [default profile](https://warewulf.org/docs/v4.6.x/release/v4.6.0.html#the-default-profile).

A major change in Warewulf 4.6 was the transition from a single monolithic `wwinit` and `generic` overlay to a modular overlay system, enabling better control over node configurations. The following overlays are included with Warewulf.

| Overlay Name         | Description                                                  | Default Enabled | Type     |
|----------------------|--------------------------------------------------------------|------------------|----------|
| wwinit               | Core initialization overlay for basic system setup           | Yes              | System   |
| wwclient             | Client service that contacts Warewulf server periodically    | Yes              | System   |
| fstab                | Configures filesystem mounts                                 | Yes              | System   |
| hostname             | Sets the node hostname                                       | Yes              | System   |
| ssh.host_keys        | Configures SSH host keys                                     | Yes              | System   |
| issue                | Sets login banner messages                                   | Yes              | System   |
| resolv               | Configures DNS resolution                                    | Yes              | System   |
| udev.netname         | Configures `udev` for network interface naming               | Yes              | System   |
| systemd.netname      | Configures `systemd` for network interface naming            | Yes              | System   |
| ifcfg                | Traditional network interface configuration                  | Yes              | System   |
| NetworkManager       | NetworkManager configuration                                 | Yes              | System   |
| debian.interfaces    | Debian-specific network configuration                        | Yes              | System   |
| wicked               | SUSE-specific network configuration                          | Yes              | System   |
| ignition             | Defines partitions and file systems on local disks           | Yes              | System   |
| hosts                | Configures `/etc/hosts` with all Warewulf nodes              | Yes              | Runtime  |
| ssh.authorized_keys  | Manages SSH authorized keys                                  | Yes              | Runtime  |
| syncuser             | Updates `/etc/passwd` and `/etc/group` with Warewulf users   | No               | Optional |
| debug                | Example overlay, not for production use                      | No               | Optional |
| localtime            | Configures the timezone of a cluster node                    | No               | Optional |
| host                 | Configuration files for the Warewulf server itself           | No               | Optional |
| netplan              | Network configuration for systems using Netplan              | No               | Optional |

You may wish to remove certain overlays or add optional ones to better fit into your environment. For instance, both `ifcfg` and `NetworkManager` are enabled by default, and you may want to choose the overlay that better aligns with your network management strategy.

### 2. Backup Existing Configuration

Although the upgrade process attempts to preserve existing configurations by creating .bak files, it's prudent to manually back up critical files using a different path or naming convention.

```bash
cp /etc/warewulf/warewulf.conf /path/to/warewulf-conf.bck 
cp /etc/warewulf/nodes.conf /path/to/nodes-conf.bck 
cp -r /var/lib/warewulf/overlays /path/to/overlays.bck  
```

### 3. Upgrade Warewulf Packages

You can install the latest Warewulf version from [CIQ Portal](https://portal.ciq.com/) as shown below.

> **Note**
> If you need access to [CIQ Portal](https://portal.ciq.com/), please send us an email at [csm-help@ciq.com](mailto:csm-help@ciq.com).

Make sure you follow [the “Getting Started” guide](https://portal.ciq.com/docs/getting-started) to familiarize yourself with Portal. Then follow [the “CIQ Depot Client” guide](https://portal.ciq.com/docs/depot-guide) to set up the necessary repository.

1. Install the depot client.

    ```bash
    sudo dnf install -y https://depot.ciq.com/public/files/depot-client/depot/depot.x86_64.rpm
    ```

2. Log in to depot.

    ```bash
    sudo depot login -u [USER STRING] -t [USER TOKEN]
    ```

3. Enable Warewulf product.

    ```bash
    sudo depot enable warewulf
    ```

4. Install Warewulf latest version.

    ```bash
    sudo dnf -y install warewulf
    ```

Alternatively, you can install Warewulf using your preferred method of installation highlighted in the [Warewulf documentation](https://warewulf.org/docs/main/server/installation.html). For example, on a system using dnf:

```bash
sudo dnf install https://github.com/warewulf/warewulf/releases/download/v4.6.1/warewulf-4.6.1-1.el9.x86_64.rpm
```

Ensure that the wwctl command reflects the new version:

```bash
# sudo wwctl version
wwctl version:  4.6.1-1
rpc version: apiPrefix:"rc1"  apiVersion:"1"  warewulfVersion:"4.6.1-1"
```

### 4. Upgrade Configuration Files

Utilize the new `wwctl upgrade` command to update configuration files:

```bash
sudo wwctl upgrade config
sudo wwctl upgrade nodes --add-defaults --replace-overlays
```

- `wwctl upgrade config`: Updates `warewulf.conf` to align with the new version's structure.
- `wwctl upgrade nodes`: Updates `nodes.conf`, adding default settings and replacing deprecated overlays.

> **Note**
> The `--replace-overlays` flag should be used only once, as it's not idempotent. This has been improved in [v4.6.1](https://warewulf.org/docs/main/release/v4.6.1.html#upgrade-fixes-and-default-behaviors), which now avoids adding the same overlay multiple times to the same node or profile. This flag replaces any reference to the `generic` or `wwinit` overlays with the new set of overlays that replace their behavior. If you do not wish to replace overlays, specify `--replace-overlays=false`.

> **Note**
> `--add-defaults` adds the new default settings to the default profile when those settings are absent. If you do not wish to add defaults, or you do not use defaults, specify `--add-defaults=false`.

### 5. Rebuild Node Images and Overlays

Optionally, after upgrading configurations, you could rebuild your node images to make sure the changes from the updated overlays and default settings are incorporated.

```bash
sudo wwctl image build <image_name>
```

Replace `<image_name>` with the appropriate image identifier.

Rebuild overlays by running

```bash
sudo wwctl overlay build
```

### 6. Verify and Test

Ensure that the updated configurations and images function as expected:

- Deploy a test node and verify successful boot and operation.

- Check logs for any errors or warnings.

- Confirm that overlays and profiles are correctly applied.

## Common Issues

### Syncuser

`syncuser` synchronizes user and group information (UIDs/GIDs) between the Warewulf server and compute node images, ensuring consistent user identities across the cluster. It reads `/etc/passwd` and `/etc/group` from both the server and node, providing a unified and conflict-free user database on every node.

#### Changes in Warewulf 4.6.0

- **No longer a default overlay:** syncuser must now be explicitly enabled; it’s not applied automatically. If `--replace-overlays=true` is used, it is included to maintain expected behavior for an existing deployment.

- **Improved duplicate handling:** Skips duplicate users/groups using a new template function.

- **Template-based:** Uses templates to generate combined `/etc/passwd` and `/etc/group` files.

#### How to Enable syncuser

**Synchronize the image:**
Use the `wwctl image syncuser your-image-name` command to update the UIDs and GIDs in your container image to match those on the server. The `wwctl image syncuser --write` parameter is now automatically enabled (specify `--write=false` to disable).

**Enable the syncuser overlay:**
Add the syncuser overlay to your node or profile configuration.

To add this to a specific node, run the following, making note of your existing overlays:

```bash
sudo wwctl node list your-node-name -a | grep RuntimeOverlay
sudo wwctl node set your-node-name --runtime-overlays=<existing-overlays>,syncuser
```

### Overlay Management

It is recommended to understand what overlays are being used and the changes that the 4.6.x versions bring, so the upgrade process goes smoothly.

Overlays have been further separated into “distribution” and “site” overlays. All overlays provided with Warewulf are “distribution” overlays and should not be modified. New overlays and modifications to distribution overlays are stored as “site” overlays. Site overlays are retained between Warewulf upgrades and take precedence over a distribution overlay of the same name.

A modified distribution overlay left behind in the new site overlay location would override the entire overlay. There have been cases where this was breaking boot, because the new `wwinit` overlay was being overridden unexpectedly.

### Defaults

At various points, Warewulf v4 has had a number of built-in default settings. These settings were once “compiled in,” and more recently were moved to a dedicated `defaults.conf` file. In v4.6.0, these defaults have been moved to the default profile and are included in `nodes.conf` for new installations.

A legacy configuration from a previous Warewulf installation can be updated to include recommended defaults using `wwctl upgrade nodes --add-defaults`. Although if your configuration is not using any defaults, it is recommended not to use the `--add-defaults` option when upgrading, as this will create a default profile with default values that may overwrite your specific configurations.

### Changes to Dracut Two-Stage Booting

Warewulf version 4.6.0 removed the dedicated `dracut.ipxe` template that was first introduced in 4.5.3. Dracut booting enhances compatibility with systems that struggle to boot large images directly. It utilizes a two-stage boot process, where a compact initramfs loads first and subsequently fetches the full node image, effectively addressing many issues associated with large image sizes. This functionality was integrated into the main `default.ipxe` template with a menu system. 

To use dracut booting in 4.6.x, you now need to set an `IPXEMenuEntry` tag instead of specifying a different template:

```shell
wwctl node set your-node-name --tagadd IPXEMenuEntry=dracut
```

The upgrade system automatically handles migration from the old dracut template to the new tag-based system. When upgrading, if a node has `ipxe: "dracut"` configured, it gets converted to use `ipxe: "default"` with the `IPXEMenuEntry: "dracut"` tag.

### Upgrade Commands

As noted at the beginning of this article, the introduction of the [upgrade command](https://warewulf.org/docs/v4.6.x/release/v4.6.0.html#upgrade) streamlines the process of ensuring that your existing configuration remains compatible with changes introduced in newer versions.

One common issue we’ve observed is the installation of version 4.6.x without first completing the upgrade steps outlined in this guide. This often leads to kernel panics and errors such as "*unable to mount root fs on unknown-block(0,0).*" Please refer to our [knowledge base article](https://kb.ciq.com/article/unable-to-mount-root-fs-on-unknown-block) for more information.

## Conclusion

Upgrading to Warewulf 4.6.x introduces several enhancements that improve cluster provisioning and management. By following this guide, administrators can ensure a seamless transition, maintaining system stability while benefiting from the latest features.

Remember, proactive backups and thorough testing are key to a successful upgrade. Embrace the improvements in Warewulf 4.6.x to optimize your cluster operations.

## References & Related Articles

[Warewulf 4.6.0 Changelog](https://warewulf.org/docs/v4.6.x/release/v4.6.0.html)  
[Warewulf 4.6.1 Changelog](https://warewulf.org/docs/v4.6.x/release/v4.6.1.html)  
[Warewulf Overlay Documentation](https://warewulf.org/docs/v4.6.x/overlays/overlays.html)


Upgrading Warewulf to Version 4.6.x


## Introduction

In the Warewulf 4.x series and higher, whenever any changes are made to `nodes.conf` or `warewulf.conf`, the `warewulfd` `systemd` service has to be restarted in order to properly reflect the changes.

## Problem

Nodes may not boot with the expected container image, despite following standard configuration and deployment steps.  
Attempts to set the container through the `wwctl node set` command and rebuild overlays do not push the new image to the node.  
Even deleting and re-adding the node using the correct container configuration does not resolve the issue.  
A node will continue to boot with an old or default container image.  
Verification using `wwctl node list -a` shows the correct container value set, however the new image is not actually being provisioned to the node.  
The issue persists even after rebuilding the overlays and restarting the node.

## Resolution

Direct edits to `nodes.conf` do not trigger a configuration reload by `warewulfd`.  As a result, the daemon continues using the old cached configuration until it is explicitly restarted.  
Any restart to `warewulfd` forces it to reload `nodes.conf` and `warewulf.conf`, after which the correct container will be applied.  

## Root cause

The root cause is due to the `nodes.conf` and `warewulf.conf` files being edited directly, instead of using the `wwctl node set` command.  

If you run `wwctl node set`, this sends a signal to `warewulfd` to reload the configuration, but editing the file manually bypasses this mechanism, so it is recommended to use `wwctl node set` instead.

### Additional troubleshooting steps for configuration changes not taking effect

- Confirm container is set for the node:  
  `sudo wwctl node set <NODE_NAME> --container=<CONTAINER_NAME>`  

- Verify the container assignment:  
  `sudo wwctl node list -a <NODE_NAME>`  

- Rebuild the node configuration:  
  `sudo wwctl configure node <NODE_NAME>`  

- Rebuild the overlays:  
  `sudo wwctl overlay build -a`  

- Reboot the node:  
  `sudo wwctl node power off <NODE_NAME>`  
  `sudo wwctl node power on <NODE_NAME>`  

- If unsuccessful, verify container is present and built:  
  `sudo wwctl container list`  
  `sudo wwctl container show <CONTAINER_NAME>`  

- Optionally, delete and re-add the node:  
  `sudo wwctl node delete <NODE_NAME>`  
  `sudo wwctl node add <NODE_NAME> --container=<CONTAINER_NAME> --ipaddr=<IP>`  

- Reconfigure and rebuild:  
  `sudo wwctl configure node <NODE_NAME>`  
  `sudo wwctl overlay build -a`  

- Set profile and container explicitly if needed:  
  `sudo wwctl node set <NODE_NAME> --profile=<PROFILE_NAME> --container=<CONTAINER_NAME>`  

- Final node reboot:  
  `sudo wwctl node power off <NODE_NAME>`  
  `sudo wwctl node power on <NODE_NAME>`  

## References & related articles

[Warewulf Cluster Nodes Documentation](https://warewulf.org/docs/main/nodes/nodes.html)


CIQ Knowledge Base

Recently updated