Upgrading Warewulf to Version 4.6.x

Introduction

Warewulf is a powerful cluster provisioning system that has undergone significant enhancements in its 4.6.x release. This guide provides a step-by-step walkthrough for upgrading your existing Warewulf 4.5.x to version 4.6.x, ensuring a smooth transition while leveraging the latest features and improvements.

Prerequisites

Before initiating the upgrade process, ensure the following:

• You are currently running Warewulf version 4.5.x.

• A complete backup of your Warewulf configuration files (warewulf.conf, nodes.conf, and overlays) is available.

• Familiarity with command-line operations and basic system administration tasks.

Instructions

1. Review the Release Notes

Familiarize yourself with the changes introduced in Warewulf 4.6.x by reviewing the official release notes. Key updates include:

• Introduction of the wwctl upgrade command for streamlined upgrades.

• Transition from "containers" to "images" terminology.

• Major changes to the syncuser option and the addition of the syncuser overlay.

• Removal of the wwctl kernel command and kernel overrides.

• Enhanced overlay management.

• The default settings moved to the default profile.

A major change in Warewulf 4.6 was the transition from a single monolithic wwinit and generic overlay to a modular overlay system, enabling better control over node configurations. The following overlays are included with Warewulf.

Overlay Name	Description	Default Enabled	Type
wwinit	Core initialization overlay for basic system setup	Yes	System
wwclient	Client service that contacts Warewulf server periodically	Yes	System
fstab	Configures filesystem mounts	Yes	System
hostname	Sets the node hostname	Yes	System
ssh.host_keys	Configures SSH host keys	Yes	System
issue	Sets login banner messages	Yes	System
resolv	Configures DNS resolution	Yes	System
udev.netname	Configures udev for network interface naming	Yes	System
systemd.netname	Configures systemd for network interface naming	Yes	System
ifcfg	Traditional network interface configuration	Yes	System
NetworkManager	NetworkManager configuration	Yes	System
debian.interfaces	Debian-specific network configuration	Yes	System
wicked	SUSE-specific network configuration	Yes	System
ignition	Defines partitions and file systems on local disks	Yes	System
hosts	Configures /etc/hosts with all Warewulf nodes	Yes	Runtime
ssh.authorized_keys	Manages SSH authorized keys	Yes	Runtime
syncuser	Updates /etc/passwd and /etc/group with Warewulf users	No	Optional
debug	Example overlay, not for production use	No	Optional
localtime	Configures the timezone of a cluster node	No	Optional
host	Configuration files for the Warewulf server itself	No	Optional
netplan	Network configuration for systems using Netplan	No	Optional

You may wish to remove certain overlays or add optional ones to better fit into your environment. For instance, both ifcfg and NetworkManager are enabled by default, and you may want to choose the overlay that better aligns with your network management strategy.

2. Backup Existing Configuration

Although the upgrade process attempts to preserve existing configurations by creating .bak files, it's prudent to manually back up critical files using a different path or naming convention.

cp /etc/warewulf/warewulf.conf /path/to/warewulf-conf.bck 
cp /etc/warewulf/nodes.conf /path/to/nodes-conf.bck 
cp -r /var/lib/warewulf/overlays /path/to/overlays.bck  

3. Upgrade Warewulf Packages

You can install the latest Warewulf version from CIQ Portal as shown below.

Note If you need access to CIQ Portal, please send us an email at csm-help@ciq.com.

Make sure you follow the “Getting Started” guide to familiarize yourself with Portal. Then follow the “CIQ Depot Client” guide to set up the necessary repository.

1. Install the depot client.

   sudo dnf install -y https://depot.ciq.com/public/files/depot-client/depot/depot.x86_64.rpm
   

2. Log in to depot.

   sudo depot login -u [USER STRING] -t [USER TOKEN]
   

3. Enable Warewulf product.

   sudo depot enable warewulf
   

4. Install Warewulf latest version.

   sudo dnf -y install warewulf
   

Alternatively, you can install Warewulf using your preferred method of installation highlighted in the Warewulf documentation. For example, on a system using dnf:

sudo dnf install https://github.com/warewulf/warewulf/releases/download/v4.6.1/warewulf-4.6.1-1.el9.x86_64.rpm

Ensure that the wwctl command reflects the new version:

# sudo wwctl version
wwctl version:  4.6.1-1
rpc version: apiPrefix:"rc1"  apiVersion:"1"  warewulfVersion:"4.6.1-1"

4. Upgrade Configuration Files

Utilize the new wwctl upgrade command to update configuration files:

sudo wwctl upgrade config
sudo wwctl upgrade nodes --add-defaults --replace-overlays

• wwctl upgrade config: Updates warewulf.conf to align with the new version's structure.
• wwctl upgrade nodes: Updates nodes.conf, adding default settings and replacing deprecated overlays.

Note The --replace-overlays flag should be used only once, as it's not idempotent. This has been improved in v4.6.1, which now avoids adding the same overlay multiple times to the same node or profile. This flag replaces any reference to the generic or wwinit overlays with the new set of overlays that replace their behavior. If you do not wish to replace overlays, specify --replace-overlays=false.

Note --add-defaults adds the new default settings to the default profile when those settings are absent. If you do not wish to add defaults, or you do not use defaults, specify --add-defaults=false.

5. Rebuild Node Images and Overlays

Optionally, after upgrading configurations, you could rebuild your node images to make sure the changes from the updated overlays and default settings are incorporated.

sudo wwctl image build <image_name>

Replace <image_name> with the appropriate image identifier.

Rebuild overlays by running

sudo wwctl overlay build

6. Verify and Test

Ensure that the updated configurations and images function as expected:

• Deploy a test node and verify successful boot and operation.

• Check logs for any errors or warnings.

• Confirm that overlays and profiles are correctly applied.

Common Issues

Syncuser

syncuser synchronizes user and group information (UIDs/GIDs) between the Warewulf server and compute node images, ensuring consistent user identities across the cluster. It reads /etc/passwd and /etc/group from both the server and node, providing a unified and conflict-free user database on every node.

Changes in Warewulf 4.6.0

• No longer a default overlay: syncuser must now be explicitly enabled; it’s not applied automatically. If --replace-overlays=true is used, it is included to maintain expected behavior for an existing deployment.

• Improved duplicate handling: Skips duplicate users/groups using a new template function.

• Template-based: Uses templates to generate combined /etc/passwd and /etc/group files.

How to Enable syncuser

Synchronize the image: Use the wwctl image syncuser your-image-name command to update the UIDs and GIDs in your container image to match those on the server. The wwctl image syncuser --write parameter is now automatically enabled (specify --write=false to disable).

Enable the syncuser overlay: Add the syncuser overlay to your node or profile configuration.

To add this to a specific node, run the following, making note of your existing overlays:

sudo wwctl node list your-node-name -a | grep RuntimeOverlay
sudo wwctl node set your-node-name --runtime-overlays=<existing-overlays>,syncuser

Overlay Management

It is recommended to understand what overlays are being used and the changes that the 4.6.x versions bring, so the upgrade process goes smoothly.

Overlays have been further separated into “distribution” and “site” overlays. All overlays provided with Warewulf are “distribution” overlays and should not be modified. New overlays and modifications to distribution overlays are stored as “site” overlays. Site overlays are retained between Warewulf upgrades and take precedence over a distribution overlay of the same name.

A modified distribution overlay left behind in the new site overlay location would override the entire overlay. There have been cases where this was breaking boot, because the new wwinit overlay was being overridden unexpectedly.

Defaults

At various points, Warewulf v4 has had a number of built-in default settings. These settings were once “compiled in,” and more recently were moved to a dedicated defaults.conf file. In v4.6.0, these defaults have been moved to the default profile and are included in nodes.conf for new installations.

A legacy configuration from a previous Warewulf installation can be updated to include recommended defaults using wwctl upgrade nodes --add-defaults. Although if your configuration is not using any defaults, it is recommended not to use the --add-defaults option when upgrading, as this will create a default profile with default values that may overwrite your specific configurations.

Changes to Dracut Two-Stage Booting

Warewulf version 4.6.0 removed the dedicated dracut.ipxe template that was first introduced in 4.5.3. Dracut booting enhances compatibility with systems that struggle to boot large images directly. It utilizes a two-stage boot process, where a compact initramfs loads first and subsequently fetches the full node image, effectively addressing many issues associated with large image sizes. This functionality was integrated into the main default.ipxe template with a menu system.

To use dracut booting in 4.6.x, you now need to set an IPXEMenuEntry tag instead of specifying a different template:

wwctl node set your-node-name --tagadd IPXEMenuEntry=dracut

The upgrade system automatically handles migration from the old dracut template to the new tag-based system. When upgrading, if a node has ipxe: "dracut" configured, it gets converted to use ipxe: "default" with the IPXEMenuEntry: "dracut" tag.

Upgrade Commands

As noted at the beginning of this article, the introduction of the upgrade command streamlines the process of ensuring that your existing configuration remains compatible with changes introduced in newer versions.

One common issue we’ve observed is the installation of version 4.6.x without first completing the upgrade steps outlined in this guide. This often leads to kernel panics and errors such as "unable to mount root fs on unknown-block(0,0)." Please refer to our knowledge base article for more information.

Conclusion

Upgrading to Warewulf 4.6.x introduces several enhancements that improve cluster provisioning and management. By following this guide, administrators can ensure a seamless transition, maintaining system stability while benefiting from the latest features.

Remember, proactive backups and thorough testing are key to a successful upgrade. Embrace the improvements in Warewulf 4.6.x to optimize your cluster operations.

References & Related Articles

Warewulf 4.6.0 Changelog
Warewulf 4.6.1 Changelog
Warewulf Overlay Documentation