Best Practices for Applying OS Updates Across Node Images

Introduction

This article outlines the best practices for applying OS updates across node images in a Warewulf environment. Following these practices helps ensure the stability and reliability of your system during updates.

Problem

Administrators often face challenges when running OS updates against node images, including the risk of breaking the image or having difficulties in reverting changes. Ensuring a smooth update process is crucial for maintaining the operational integrity of the system.

Symptoms

• Inconsistent node behavior post-update.
• Node failures or inability to boot.
• Difficulty in reverting to a previous stable state after updates.

Resolution

To minimize risks and ensure a smooth update process, follow these best practices in order of preference:

Best practices when updating images

When planning OS updates in a Warewulf environment, here are three approaches you may consider: the primary method builds a new image using an updated Containerfile which can be run via a CI/CD pipeline, the secondary method copies the current image to a new image for updating, and the final method directly updates the existing image, which is riskier as it will not retain the original, working image.

Primary method

1. Update the Containerfile: Apply updates to the Containerfile used to build the node image.
2. Build a New Image: Construct a new image from the updated Containerfile.
3. Import into Warewulf: Import the newly built image into Warewulf.
4. Gradual Migration: Gradually migrate nodes to the new image to monitor and ensure stability.

Secondary method

1. Copy Current Image: Duplicate the current node image and assign a new name to the copy.
2. Apply Updates: Implement the necessary updates on the copied image.
3. Gradual Migration: Gradually transition nodes to the updated image to ensure stability and functionality.

Tertiary method

1. Direct Updates: Apply updates directly to the current node image. This method is riskier and should only be used if the first two methods are not feasible.
2. Backup: Ensure there is a backup of the current image before proceeding with updates. This allows for a rollback if the update process fails or causes issues.

Adding an image to nodes

When creating a new image, you have a couple options for applying the image within your environment, primarily at either the profile level or the node level. The configuration set directly on a node, overrides the configuration from a profile. For example, you can configure a node to use a different image than what's configured on the default profile.

Updating individual nodes

In this example, you will select a specific test node, n8, to upgrade to the newly built 8.10 Rocky Linux image. Please note that in Warewulf 4.5.8 and prior, the flag to add or edit a container image was --container. In 4.6.0 this was changed to --image to better reflect its purpose. However, the --container flag still works for compatibility.

$ wwctl node set n8 --image=rockylinux-8.10 

$ wwctl node list n8 --all
  NODE  FIELD                     PROFILE     VALUE
  n8    Id                        --          n8
  n8    Comment                   default     This profile is automatically included for each node
  n8    ImageName                 SUPERSEDED  rockylinux-8.10
  n8    Ipxe                      --          (default)
  n8    RuntimeOverlay            --          (generic)
  n8    SystemOverlay             --          (wwinit, [...])
  n8    Root                      --          (initramfs)
  n8    Init                      --          (/sbin/init)
  n8    Kernel.Args               --          (quiet crashkernel=no vga=791 net.naming-scheme=v238)
  n8    Profiles                  --          default,cluster1
  n8    PrimaryNetDev             --          (default)
  n8    NetDevs[default].Type     --          (ethernet)
  n8    NetDevs[default].OnBoot   --          (true)
  n8    NetDevs[default].Device   cluster1    eno1
  n8    NetDevs[default].Netmask  cluster1    255.255.255.0
  n8    NetDevs[default].Gateway  cluster1    10.0.0.3
  n8    NetDevs[default].Primary  --          (true)

The ImageName field for the n8 node is now set to rockylinux-8.10. The SUPERSEDED value in the profile column indicates that a node value is replacing a profile value typically set from a profile. This allows you to test the new image on an individual server before rolling it out to a wider selection of nodes.

Updating a profile

You can also create a new profile. In this example, you will call it dev so it can be rolled out to multiple nodes at once. This is useful, particularly when testing, when you have multiple generations of hardware or configurations of hardware that need to be tested and validated to work with the new image. You will first start by creating the profile if it has not been created already.

$ wwctl profile add dev --comment="Development/testing environment"

$ wwctl profile list
  PROFILE NAME  COMMENT/DESCRIPTION
  dev           Development/testing environment
  default       This profile is automatically included for each node

Once this new dev profile is set, you can add information like you would on a node. In this case, apply a newer rockylinux-8.10v2 image to this profile.

wwctl profile set dev --image=rockylinux-8.10v2

You can now add your node n8 to this profile

wwctl node set n8 --profile=default,dev

While you can configure a node with one profile, one of the most powerful features of Warewulf is the ability to combine profiles. If two profiles set the same field, such as the image field, the right-most profile in the node’s list takes precedence. In the example above, the image from dev will take precedence over the image in default. Reminder: values set directly on nodes will take precedence over profile field values. If you follow this guide, n8 was set to use rockylinux-8.10 as its image directly and as a result, this value overrides anything within the profiles. If you unset this value on the node, you will see n8 use the rockylinux-8.10v2 image:

wwctl node set n8 --image=UNSET

$ wwctl node list n8 -a
NODE  FIELD                        PROFILE  VALUE
----  -----                        -------  -----
n8    Profiles                     --       default,dev
n8    Comment                      dev      Development/testing environment
n8    ImageName                    dev      rockylinux-8.10v2
[...]

Root Cause

The risk of breaking the image or encountering issues increases significantly when updates are applied directly to the current image. Using a Containerfile to build and update images or duplicating the current image before applying updates helps maintain stability and provides a clear rollback path.

Notes

• Always ensure there is a backup of the current image before applying any updates.
• Gradual migration allows for monitoring and quick identification of issues, reducing the impact on the overall system.

References & related articles

Warewulf Profile Set Documentation
Warewulf Node Set Documentation
Warewulf Profiles Documentation
Warewulf Images Documentation