Warewulf Restart Required After Configuration or Image Changes

Introduction

In the Warewulf 4.x series and higher, whenever any changes are made to nodes.conf or warewulf.conf, the warewulfd systemd service has to be reloaded or restarted to properly reflect the changes. The same applies when a node's or profile's assigned image changes outside of wwctl, for example when an image is renamed and the profile is updated to point to the new name by an external process. If that change doesn't reach warewulfd, the daemon keeps using the cached image assignment and nodes can fail to boot, even though wwctl shows the correct image and kernel.

Problem

• Nodes may not boot with the expected container image, despite following standard configuration and deployment steps.
• Attempts to set the container through the wwctl node set command and rebuild overlays do not push the new image to the node.
• Even deleting and re-adding the node using the correct container configuration does not resolve the issue.
• A node continues to boot with an old or default container image.
• Verification using wwctl node list -a shows the correct container value set, however the new image is not actually being provisioned to the node.
• The issue persists even after rebuilding the overlays and restarting the node.

After renaming or reassigning an image

• A node that booted fine before fails to boot after an image is renamed, or after a node's or profile's image assignment changes outside of wwctl.
• iPXE reports Could not boot image: no such file or directory.
• wwctl image list -l and wwctl image kernels show the node's current image with the correct kernel present and set as the default.
• Rebuilding the image content or overlays doesn't help, because the image itself is intact and its files are served correctly. The problem is the image name the daemon has cached for the node.

Resolution

Direct edits to nodes.conf do not trigger a configuration reload by warewulfd. As a result, the daemon continues using the old cached configuration until it is explicitly restarted. Any restart to warewulfd forces it to reload nodes.conf and warewulf.conf, after which the correct container is applied.

Confirming a stale node database

warewulfd resolves a node's kernel from the image assigned in its cached node database, then reads the kernel files from that image on disk. If the cached image assignment is stale, for example it still points at an image that was renamed away, the lookup finds no kernel and the daemon logs the following (a sosreport with the warewulf-sos plugin captures these logs):

ERROR : No kernel found for node compute01
ERROR : No resource selected

Reload or restart the daemon so it re-reads nodes.conf and picks up the current image assignment, then boot the node again:

sudo systemctl reload warewulfd

A full restart (sudo systemctl restart warewulfd) has the same effect. Note that a plain image content rebuild does not require this: warewulfd reads the kernel from the image on disk on every request, so a rebuilt image under the same name is picked up automatically. A reload is only needed when the node or profile image assignment changed without going through wwctl. Changes made with wwctl node set or wwctl profile set reload the daemon automatically; direct edits to nodes.conf and warewulf.conf, or changes made by external automation, do not.

Root cause

The issue is caused by the nodes.conf and warewulf.conf files being edited directly, instead of using the wwctl node set command.

If you run wwctl node set, this sends a signal to warewulfd to reload the configuration, but editing the file manually bypasses this mechanism, so it is recommended to use wwctl node set instead.

Additional troubleshooting steps for configuration changes not taking effect

• Confirm container is set for the node: sudo wwctl node set <NODE_NAME> --container=<CONTAINER_NAME>

• Verify the container assignment: sudo wwctl node list -a <NODE_NAME>

• Rebuild the node configuration: sudo wwctl configure node <NODE_NAME>

• Rebuild the overlays: sudo wwctl overlay build -a

• Reboot the node: sudo wwctl node power off <NODE_NAME> sudo wwctl node power on <NODE_NAME>

• If unsuccessful, verify container is present and built: sudo wwctl container list sudo wwctl container show <CONTAINER_NAME>

• Optionally, delete and re-add the node: sudo wwctl node delete <NODE_NAME> sudo wwctl node add <NODE_NAME> --container=<CONTAINER_NAME> --ipaddr=<IP>

• Reconfigure and rebuild: sudo wwctl configure node <NODE_NAME> sudo wwctl overlay build -a

• Set profile and container explicitly if needed: sudo wwctl node set <NODE_NAME> --profile=<PROFILE_NAME> --container=<CONTAINER_NAME>

• Final node reboot: sudo wwctl node power off <NODE_NAME> sudo wwctl node power on <NODE_NAME>

References & related articles

Warewulf Cluster Nodes Documentation
Best Practices for Applying OS Updates Across Node Images
Troubleshooting PXE Booting and Node Provisioning