Unable to Mount Root FS After Warewulf Upgrade

Introduction

When upgrading from Warewulf 4.5.8 to Warewulf 4.6.0 and above, there are significant changes to the configuration. We recommend reading the release notes to better understand the differences. Failure to follow the instructions will result in an environment where your compute nodes may be unable to boot.

Problem

When booting a compute node within a cluster, you encounter a Kernel Panic stating the system is Unable to mount root fs on unknown-block(0,0).

Root Cause

In Warewulf 4.5.8 and earlier, the necessary components to boot a compute node were present in one single overlay, wwinit. In Warewulf 4.6.0 and above, the components were separated into multiple overlays. The Warewulf config needs to be upgraded to reference these new overlays.

Resolution

Ensure that you read the release notes to determine what options apply for your environment. Before running anything, we recommend creating a backup of your warewulf host node. Typically you will need to run two commands:

wwctl upgrade config
wwctl upgrade nodes --add-defaults --replace-overlays

The first command will upgrade your warewulf.conf file and the second will upgrade your nodes.conf with the necessary changes to the overlays and the default profile. Both of these will attempt to update their respective configuration file in-place, retaining a copy of the previous version. You can read more about these flags below:

• --add-defaults: This adds the new default settings to the default profile when those settings are absent. If you do not wish to add defaults, specify --add-defaults=false.
• --replace-overlays: This flag replaces any reference to the generic or wwinit overlays with the new set of overlays that replace their behavior. If you do not wish to replace overlays, specify --replace-overlays=false.

Warning: The upgrade with the --replace-overlays flag should only be run once.

Once you finish the upgrade process, you can run wwctl overlay build and attempt to boot your compute nodes once again.

Notes

You may also need to rebuild your image after the upgrade if you see a No such file or directory error message when attempting to boot. You can run wwctl image build <image name> to build the image if it is necessary.

References & related articles

Warewulf 4.6.0 Changelog
Warewulf 4.6.1 Changelog
Warewulf Overlay Documentation