Preventing Warewulf Ignition From Wiping the Wrong Disk With Stable Device Paths
Introduction
Warewulf can hand disk partitioning and filesystem creation off to Ignition on Rocky Linux nodes. The disk to operate on is named in the node's Ignition disk configuration. When that configuration points at a kernel device name such as /dev/sda, and the node has more than one device in the sd* namespace, the target can shift from one boot to the next. This article covers why that happens and how to make the target stable by pointing the configuration at a persistent identifier under /dev/disk/by-id/ or /dev/disk/by-path/. It was written against Warewulf 4.6.x and also applies to 4.5.x, which introduced Ignition-based disk provisioning as a preview feature. Releases before 4.5 do not provision disks through Ignition.
Problem
Warewulf nodes boot over the network, and Ignition initializes their local disks during the boot process. When a node has more than one block device in the sd* namespace (for example a data SSD alongside a second disk, or a USB device left connected), the kernel does not guarantee which physical device receives the sda name on any given boot. An Ignition disk configuration that wipes and rewrites the partition table on /dev/sda therefore acts on whichever device happens to be named sda at that boot. After hardware maintenance or a cold power cycle, that can be a different device than the one you intended.
The result is data loss on the device that unexpectedly took the sda name, and a node whose local storage is not laid out the way the configuration intended.
Symptoms
After maintenance or a power cycle, a node's local storage is not what was configured: the partition table on the intended disk is gone, or a second device that should have been left alone has been wiped. Both devices are visible in the same sd* namespace, and which one is sda is not consistent across boots:
$ lsblk -o NAME,SIZE,TRAN,MODEL
NAME SIZE TRAN MODEL
sda 28.7G usb USB Flash Drive
sdb 931.5G sata SATA SSD 1TB
The Ignition disk configuration on the affected nodes targets a kernel device name rather than a stable path:
$ wwctl node list -a node001 | grep -i disk
node001 Disks[/dev/sda].WipeTable -- true
ℹ️ NOTE Kernel arguments such as rd.ignoredisk= do not prevent the wipe. Ignition writes to the exact device named in its own configuration, so influencing what the kernel enumerates at boot does not change Ignition's target.
Resolution
The durable fix is to name the SSD by a stable identifier so Ignition always operates on the same physical device regardless of enumeration order. Do this in two stages: stop the destructive wipe immediately, then update the disk configuration to point at a stable path and validate on one node before rolling out.
Stop the wipe before any further reboots
While you stage the permanent fix, disable the table wipe so no additional nodes have the wrong device erased. For a single node:
wwctl node set node001 --diskname /dev/sda --diskwipe=false
wwctl overlay build node001
Apply this to every affected node before it reboots.
Find the stable identifier of the SSD
On the node, list the persistent aliases and pick the one that belongs to the SSD, not the USB drive. Identifiers under /dev/disk/by-id/ (serial or WWN based) follow the physical drive and are the most durable choice. Identifiers under /dev/disk/by-path/ are tied to the bus location and are a good alternative when a by-id alias is not available:
ls -l /dev/disk/by-id/
ls -l /dev/disk/by-path/
Confirm the alias resolves to the SSD (for example sdb above) and not the USB drive before using it.
Point the disk configuration at the stable path
Set the disk name to the stable identifier of the SSD, re-enable the wipe, rebuild the overlay, and reboot the node. Substitute the identifier you confirmed for the SSD:
wwctl node set node001 --diskname /dev/disk/by-id/ata-<model>_<serial> --diskwipe=true
wwctl overlay build node001
The resulting configuration names the SSD by its persistent alias:
Disks:
/dev/disk/by-id/ata-<model>_<serial>:
wipe_table: true
partitions:
root:
number: 1
should_exist: true
Validate on one node, then roll out
Pick a single node, reboot it several times, and confirm Ignition consistently targets the intended SSD and leaves the other devices untouched. Once the behavior holds across a couple of boots, apply the same change to the remaining nodes.
Root cause
The sd* device names are assigned by the kernel in the order devices are discovered, which is not stable across boots. When more than one device lives in that namespace, any of them can become /dev/sda on a given boot. Because the Ignition disk configuration named /dev/sda directly, wipe_table: true was applied to whichever device won that name at boot time. This is a configuration issue rather than a Warewulf or Ignition defect: the fix is to reference the drive by an identifier that is bound to the physical device rather than to enumeration order.
Notes
- Disk wipe and repartition operations are destructive and erase all data on the targeted device. Confirm the stable identifier resolves to the intended SSD before enabling the wipe, and validate on a single non-critical node with a maintenance window in place before rolling the change out to a fleet.
- Reverting a node's kernel arguments to the profile default before testing avoids working from a modified baseline. Setting
--kernelargson a node replaces the entire kernel argument string rather than appending to it.
References & related articles
How to Setup Ignition with Warewulf on Rocky Linux 8.x