Best Practices for Applying OS Updates Across Node Images

Introduction

This article outlines the best practices for applying operating system (OS) updates across node images in a Warewulf environment. Following these practices helps ensure the stability and reliability of your system during updates.

Problem

Administrators often face challenges when applying OS updates across node images, including the risk of breaking the image or having difficulties in reverting changes. Ensuring a smooth update process is crucial for maintaining the operational integrity of the system.

Symptoms

• Inconsistent node behavior post-update.
• Node failures or inability to boot.
• Difficulty in reverting to a previous stable state after updates.

Resolution

To minimize risks and ensure a smooth update process, follow these best practices in order of preference:

Best practices when updating images

When planning OS updates in a Warewulf environment, here are three approaches you may consider. The primary method builds a new image using an updated Containerfile which can be run through to a CI/CD pipeline, the secondary copies the current image to a new image for updating, and the final method directly updates the existing image, which is riskier as it will not retain the original, working image.

Primary

1. Update the Containerfile: Apply updates to the Containerfile used to build the node image.
2. Build a New Image: Construct a new image from the updated Containerfile.
3. Import into Warewulf: Import the newly built image into Warewulf.
4. Gradual Migration: Gradually migrate nodes to the new image to monitor and ensure stability.

Secondary

1. Copy Current Image: Duplicate the current node image and assign a new name to the copy.
2. Apply Updates: Implement the necessary updates on the copied image.
3. Gradual Migration: Gradually transition nodes to the updated image to ensure stability and functionality.

Tertiary

1. Direct Updates: Apply updates directly to the current node image. This method is riskier and should only be used if the first two methods are not feasible.
2. Backup: Ensure there is a backup of the current image before proceeding with updates. This allows for a rollback if the update process fails or causes issues.

Adding image to nodes

When creating a new container, you have a couple options for applying it within your environment, primarily at either the profile level or the node level. Configuration set directly on a node overrides configuration from a profile. For example, we can configure a node to use a different container image than what's configured on the default profile.

Updating individual nodes

In this example, we will select a specific test node, n8, to upgrade to newly built 8.10 Rocky Linux image. Please note that in Warewulf 4.5.8 and prior, the flag to add or edit a container image was --container. In 4.6.0 this was changed to --image to better reflect its purpose. However, the --container flag still works for compatibility.

$ wwctl node set n8 --container=rockylinux-8.10 

$ wwctl node list n8 --all
  NODE  FIELD                     PROFILE     VALUE
  n8    Id                        --          n8
  n8    Comment                   default     This profile is automatically included for each node
  n8    ImageName                 SUPERSEDED  rockylinux-8.10
  n8    Ipxe                      --          (default)
  n8    RuntimeOverlay            --          (generic)
  n8    SystemOverlay             --          (wwinit, [...])
  n8    Root                      --          (initramfs)
  n8    Init                      --          (/sbin/init)
  n8    Kernel.Args               --          (quiet crashkernel=no vga=791 net.naming-scheme=v238)
  n8    Profiles                  --          default,cluster1
  n8    PrimaryNetDev             --          (default)
  n8    NetDevs[default].Type     --          (ethernet)
  n8    NetDevs[default].OnBoot   --          (true)
  n8    NetDevs[default].Device   cluster1    eno1
  n8    NetDevs[default].Netmask  cluster1    255.255.255.0
  n8    NetDevs[default].Gateway  cluster1    10.0.0.3
  n8    NetDevs[default].Primary  --          (true)

The ImageName field for the n8 node is now set to rockylinux-8.10. The SUPERSEDED value in the profile column indicates that a node value is replacing a profile value typically set from a profile. This allows you to test the new image on an individual server before rolling it out to a wider selection of nodes.

Updating a profile

We can also create a new profile, in our example we will call it dev, to roll this out to multiple nodes at once. This is useful, particularly when testing, when you have multiple generations of hardware or configurations of hardware that need to be tested and validated to work with the new image. We will first start by creating the profile if it has not been created already.

$ wwctl profile add dev --comment="Development/testing environment"

$ wwctl profile list
  PROFILE NAME  COMMENT/DESCRIPTION
  dev           Development/testing environment
  default       This profile is automatically included for each node

Once this new dev profile is set, we can add information like we would on a node, In our case, lets apply a newer rockylinux-8.10v2 image to this profile.

wwctl profile set dev --container=rockylinux-8.10v2

We can now add our node n8 to this profile

wwctl node set n8 --profile=default,dev

While you can configure a node with one profile, one of the most powerful features of Warewulf is the ability to combine profiles. If two profiles set the same field such as the image field, the right-most profile in the node’s list takes precedence. In our example above, the image from dev will take precedence over the image in default. Reminder, values set directly on nodes will take precedence over profile field values. If you follow this guide, n8 was set to use rockylinux-8.10 as its image directly and as a result, this value overrides anything within the profiles. If we unset this value on the node, we will see n8 use the rockylinux-8.10v2 image:

wwctl node set n8 --container=UNSET

$ wwctl node list n8 -a
NODE  FIELD                        PROFILE  VALUE
----  -----                        -------  -----
n8    Profiles                     --       default,dev
n8    Comment                      dev      Development/testing environment
n8    ImageName                    dev      rockylinux-8.10v2
[...]

Root Cause

The risk of breaking the image or encountering issues increases significantly when updates are applied directly to the current image. Using a Containerfile to build and update images or duplicating the current image before applying updates helps maintain stability and provides a clear rollback path.

Notes

• Always ensure there is a backup of the current image before applying any updates.
• Gradual migration allows for monitoring and quick identification of issues, reducing the impact on the overall system.

References & related articles

Warewulf Profile Set Documentation
Warewulf Node Set Documentation
Warewulf Profiles Documentation
Warewulf Images Documentation