Troubleshooting Two-Stage (Dracut) Boot Failures in Warewulf

warewulf dracut troubleshooting boot two-stage boot infiniband provisioning dhcp

Stephen Simpson
Senior Customer Support Engineer

Jul 20, 2026

Introduction

This article helps you diagnose compute nodes that fail to boot from a Warewulf two-stage (dracut) image. It is organized by symptom: use the triage table in the Resolution section to match the error you are seeing to its fix. It covers the common dracut: FATAL: Don't know how to handle 'root=wwinit:initramfs' error along with the other failures that can surface with two-stage boot: no network in the initramfs, a drop to the dracut emergency shell, and a machine-id permission that breaks NetworkManager.

This applies to Warewulf 4.6 and above, with Rocky Linux 8, 9, and 10 images. The general flow is the same across those releases; a couple of steps differ by Rocky version, and those differences are called out where they apply. If you have not yet enabled two-stage booting on your image, start with Enabling Two-Stage Booting with Dracut on an Existing Warewulf Image, then return here if the node still fails to boot.

Problem

Two-stage booting loads a small dracut initramfs first, and the kernel then fetches the full system image and its overlays from the Warewulf server over the network. Each of those handoffs can fail independently, so a single boot failure often has more than one cause. The initial error usually points at dracut, but the underlying cause is frequently a missing dracut module or a file permission that dracut re-applies on every rebuild.

Resolution

Match the symptom you are seeing to the table below, then go to the matching section for the fix. A single boot failure often has more than one cause, so if more than one row applies, work through each section in the order listed.

Symptom	Likely cause	Fix
`dracut: FATAL: Don't know how to handle 'root=wwinit:initramfs'`	The `wwinit` dracut module is missing from the initramfs	Install the wwinit dracut module
Node hangs for several minutes, then `gzip`/`cpio` errors and a NetworkManager timeout	No working network interface in the initramfs (missing driver)	Add network drivers to the initramfs
Same `gzip`/`cpio` errors on a Rocky Linux 8 image, but only on large images or only on some nodes, and a small image boots on the same node	The provisioning DHCP lease expires mid-download and the node loses its address	Raise the provisioning DHCP lease time
Node drops to a dracut emergency shell	An `/etc/fstab` entry cannot be mounted from the initramfs	Install mdadm for RAID-backed mounts
`Failed to start D-Bus`, with `machine-id` at `0640` (Rocky Linux 9 and 10)	dracut builds `machine-id` with permissions `dbus-broker` cannot read	Correct machine-id permissions

Each fix assumes you replace <image> with your image name (for example rockylinux-9) and <node> with the affected node. Match the warewulf-dracut version to your Warewulf server version, and match the RPM's EL suffix (el8, el9, or el10) to the image's Rocky Linux major version.

⚠️ WARNING These fixes rebuild the node image and its initramfs. Back up or copy your current image before making changes so you can revert to a known-good state, and apply changes during a maintenance window since affected nodes will need to reboot.

Install the wwinit dracut module

The node halts at the very start of the initramfs with:

dracut: FATAL: Don't know how to handle 'root=wwinit:initramfs'

This means the initramfs was not built with the wwinit module, which is provided by the warewulf-dracut package. Install the package and regenerate the initramfs:

wwctl image exec --build=false <image> -- /usr/bin/dnf --assumeyes install https://github.com/warewulf/warewulf/releases/download/v4.6.3/warewulf-dracut-4.6.3-1.el9.noarch.rpm
wwctl image exec <image> -- /usr/bin/dracut --force --no-hostonly --add wwinit --regenerate-all

Replace the warewulf-dracut version to match your Warewulf server version, and the el9 suffix to match the image's Rocky Linux major version (el8, el9, or el10).

For the full walkthrough of enabling two-stage boot, including the node tag that selects the dracut boot entry, see Enabling Two-Stage Booting with Dracut.

Add network drivers to the initramfs

Once dracut is handling wwinit, the node hangs for several minutes and then fails with gzip/cpio errors and a NetworkManager timeout (the wwinit module sets a 300-second nm-online wait). These are the signature of an empty network fetch: with no working interface in the initramfs, the system-image download returns nothing, and the curl | gzip -d | cpio pipeline the module uses to unpack each stage has nothing to work on.

This usually means the initramfs is missing the driver for the boot interface. Nodes that provision over InfiniBand need the rdma dracut module, which comes from the rdma-core package. The interface firmware handles the initial iPXE fetch, but once the kernel takes over, the initramfs needs those modules to bring the link up and download the system image.

wwctl image exec --build=false <image> -- /usr/bin/dnf --assumeyes install rdma-core
wwctl image exec <image> -- /usr/bin/dracut --force --no-hostonly --add "wwinit rdma" --regenerate-all

ℹ️ NOTE When using wwctl image exec <image> -- ... you must give the full binary path (for example /usr/bin/bash, not bash), otherwise the command returns ERROR: no such file or directory.

You can confirm what made it into the initramfs, and compare against a known-good image:

wwctl image exec --build=false <image> -- /usr/bin/bash -c 'lsinitrd /boot/initramfs-*.img | grep -iE "mlx|ib_|rdma|wwinit"'

If the fetch clearly gets partway before failing, or the same image fails only on some nodes while a smaller image boots, the interface is not missing. On a Rocky Linux 8 image, skip to Raise the provisioning DHCP lease time.

Raise the provisioning DHCP lease time (Rocky Linux 8)

ℹ️ NOTE This failure affects Rocky Linux 8 images only. Rocky Linux 9 and 10 images are not affected: their initramfs runs NetworkManager as a persistent service that renews the DHCP lease throughout the boot, so the provisioning address does not expire during the download. On Rocky Linux 8 the initramfs configures the network and then exits, leaving nothing to renew the lease, which is what exposes this issue.

This failure looks almost identical to the missing-driver case above, the node hangs and then fails with gzip/cpio errors, but it has two distinguishing tells. The transfer gets partway first (the small system overlay loads, and you often see a cpio block count on the console before the hang), and it is size or node dependent: a small image boots on the same node while a larger image fails, or the same image fails on some nodes but not others.

The cause is the provisioning DHCP lease expiring during the download. On a Rocky Linux 8 image, the initramfs NetworkManager runs in configure-and-quit mode: it acquires a DHCP lease, applies the address to the interface, and exits. Nothing renews the lease for the rest of the boot. Warewulf's provisioning DHCP pool hands out a short lease (120 seconds by default), and the kernel stamps the DHCP address with that lease as its valid lifetime. When the lifetime reaches zero, the kernel removes the address from the interface. If the system-image download takes longer than the lease, which happens with a large image or a slower node or link, the node loses its address mid-transfer, the download stalls, and once the stalled TCP connection finally times out the curl | gzip -d | cpio pipeline reports gzip: stdin: unexpected end of file and cpio: premature end of file. The small overlay stage succeeds because it finishes well inside the lease window.

You can confirm it from a dracut shell (see Inspect a failing node from a dracut shell) by checking the boot interface's address. Use ip a to find the interface name if you are unsure, then:

ip -4 addr show dev <interface>

The DHCP address carries a valid_lft that counts down from the lease time, and the inet line disappears once it reaches zero. The download then fails some minutes later, when the stalled TCP connection times out, which is why the console goes quiet for a while before the gzip/cpio errors appear.

The fix is to raise max-lease-time in the Warewulf host overlay DHCP template above the worst-case download time. Edit the template:

wwctl overlay edit host etc/dhcp/dhcpd.conf.ww

Find the max-lease-time 120; line, raise it to a value comfortably larger than your longest provisioning download (for example max-lease-time 3600;), and save. If your template has been customized and has no max-lease-time 120; line, run grep max-lease-time /etc/dhcp/dhcpd.conf first to find the active value, then change the matching line in the template. Then regenerate and apply the DHCP configuration:

wwctl configure dhcp
grep max-lease-time /etc/dhcp/dhcpd.conf

The rendered /etc/dhcp/dhcpd.conf should show the new value. Reboot the affected node; it should now finish the image download and boot. A larger lease only affects the provisioning window, since the node replaces the DHCP address with its permanent configuration once the image boots. Warewulf saves the edited template as a site copy of the overlay, so the change survives package updates and the packaged template is left untouched; to revert, remove the site copy and run wwctl configure dhcp again.

ℹ️ NOTE Switching the DHCP configuration to static reservations does not fix this on its own. A fixed-address reservation still hands out the same short lease, so the address still expires mid-download. Raise max-lease-time regardless of whether you provision from a dynamic pool or static reservations. Because a longer lease holds each provisioning address longer before it can be reused, confirm the pool is large enough for the number of nodes you provision at once (see DHCP best practices).

Install mdadm for RAID-backed mounts

If the node drops to a dracut emergency shell, check whether /etc/fstab in the image mounts arrays or volumes that the initramfs cannot assemble. A common cause is fstab entries for RAID arrays without mdadm installed in the image. Install it, and consider adding nofail to non-essential fstab entries so a missing mount does not block boot:

wwctl image exec <image> -- /usr/bin/dnf --assumeyes install mdadm

Installing mdadm in the image is enough when the array is mounted by the booted system after pivot. If the array has to be assembled inside the initramfs itself (an early mount required to boot), also add the mdraid dracut module and rebuild, as in the sections above:

wwctl image exec <image> -- /usr/bin/dracut --force --no-hostonly --add mdraid --regenerate-all

Correct machine-id permissions (Rocky Linux 9 and 10)

This applies to Rocky Linux 9 and 10, which use dbus-broker as the default D-Bus implementation. The node fails with:

[FAILED] Failed to start D-Bus System Message Bus

and journalctl -u dbus-broker.service shows:

ERROR launcher_run_child @ ../src/launch/launcher.c +325: Permission denied

The cause is /etc/machine-id being built into the initramfs with 0640 instead of 0444. You can confirm it from a dracut shell (see below) or with lsinitrd:

# ls -laZ /etc/machine-id
-rw-r----- ... /etc/machine-id      # wrong (0640)
-r--r--r-- ... /etc/machine-id      # correct (0444)

dbus-broker cannot read machine-id, so D-Bus fails, NetworkManager fails, and the network fetch never happens. Setting the permission on the file inside the image is not enough, because dracut re-applies the wrong permission every time it assembles the initramfs.

ℹ️ NOTE Rocky Linux 8 uses dbus-daemon rather than dbus-broker, so it does not hit this permission failure. On Rocky Linux 8 the machine-id problem instead shows up as dracut failing to generate the initramfs at all, which is resolved by removing /etc/machine-id from the image before running dracut (wwctl image exec <image> -- /usr/bin/rm -f /etc/machine-id). See Enabling Two-Stage Booting with Dracut for that variant.

The reliable fix on Rocky Linux 9 and 10 is a small custom dracut module that corrects the permission after assembly. Open a shell in the image:

wwctl image shell <image>

Inside the image shell, create the module:

mkdir -p /usr/lib/dracut/modules.d/99fix-machineid
cat > /usr/lib/dracut/modules.d/99fix-machineid/module-setup.sh << 'ENDOFFILE'
#!/bin/bash
check() { return 0; }
depends() { return 0; }
install() {
    [ -e "$initdir/etc/machine-id" ] && chmod 444 "$initdir/etc/machine-id"
}
ENDOFFILE
chmod 755 /usr/lib/dracut/modules.d/99fix-machineid/module-setup.sh
dracut --force --no-hostonly --add "wwinit rdma" --regenerate-all

Keep the --add module list consistent with what your nodes need. Include rdma only if the nodes boot over InfiniBand; on Ethernet-only nodes, use --add wwinit. The fix-machineid module does not need to be listed in --add, because its check() returns 0, dracut includes it automatically on every rebuild.

Verify the permission is now 0444 before leaving the image shell, then exit:

lsinitrd /boot/initramfs-*.img | grep machine-id
exit

Inspect a failing node from a dracut shell

If you need to inspect a failing node directly rather than match a symptom above, set it to drop into a dracut shell and rebuild its overlay:

wwctl node set <node> --kernelargs rd.shell
wwctl overlay build

From that shell, these checks isolate the failures above:

ls -laZ /etc/machine-id
journalctl -u dbus-broker.service --no-pager
journalctl -u nm-initrd.service --no-pager
ip a

Remove the rd.shell kernel argument once you are done diagnosing.

Root Cause

The root=wwinit:initramfs error is a missing wwinit dracut module (no warewulf-dracut package or an initramfs that was not rebuilt with it). Beyond that, the remaining failures share a single theme: the initramfs cannot reach the network to download the system image. That can be because the driver for the boot interface was never included (InfiniBand needs the rdma module), because the node dropped to an emergency shell on an unmountable fstab entry, or because machine-id was built into the initramfs with 0640 permissions. In the last case, on Rocky Linux 9 and 10, dbus-broker cannot read machine-id, D-Bus fails to start, NetworkManager depends on D-Bus and never brings up the interface, and the image download fails with no obvious network error. Dracut re-applies the wrong machine-id permission during initramfs assembly regardless of the permission set on the file in the image, which is why a filesystem-level chmod alone does not stick and the custom dracut module is required. Rocky Linux 8 uses dbus-daemon and does not hit this, but has its own machine-id quirk noted under Correct machine-id permissions above.

The DHCP lease-expiry case is a related but distinct failure, and it is specific to Rocky Linux 8 images. Here the network does come up and the download begins, so it is not a connectivity problem. On Rocky Linux 8 the initramfs NetworkManager configures the interface and exits without leaving anything to renew the lease, and the kernel removes the address when the lease's valid lifetime ends. On a large image whose download outlives the short provisioning lease, the address disappears mid-transfer and the truncated stream surfaces as the same gzip/cpio errors, which is why raising the lease time rather than changing the image or the drivers is what resolves it. Rocky Linux 9 and 10 images do not hit this, because their initramfs keeps NetworkManager running to renew the lease for the duration of the boot.

Notes

The following were tried against the machine-id failure and did not resolve it, so they are worth skipping:

Adding After=dbus.service via an nm-initrd.service drop-in
Manually symlinking dbus.socket into sockets.target.wants
Omitting NetworkManager to fall back to legacy networking. The wwinit module depends on the network dracut module, and on current Rocky Linux releases the default backend for network is NetworkManager, so removing it leaves the initramfs with no way to bring the interface up
rpm --setperms on the filesystem or systemd packages
Truncating or removing /etc/machine-id

All of these were ineffective because dracut re-applied the wrong machine-id permission on every rebuild. Correcting it after assembly with the custom dracut module described above is what holds.

Two unrelated issues can surface on the same nodes and are worth ruling out separately: an onboard NIC stuck in firmware recovery mode (the interface reports a 00:00:00:00:00:00 MAC and logs Firmware recovery mode detected), which is a hardware/NVM problem, and directories under an image's /opt appearing empty when /opt is NFS-mounted from the Warewulf server, since the mount hides the image contents.

Enabling Two-Stage Booting with Dracut on an Existing Warewulf Image
Troubleshooting PXE Booting and Node Provisioning
DHCP Best Practices
Warewulf Documentation - Booting with Dracut
Warewulf Documentation - Image Management
Current Warewulf Releases