Two-Stage Dracut Boot Over InfiniBand (IPoIB)

Introduction

Warewulf two-stage dracut boot needs extra setup when nodes provision over InfiniBand (IPoIB) instead of Ethernet. The standard two-stage boot setup documented in Enabling Two-Stage Booting with Dracut is a prerequisite.

IPoIB boot adds complexity because the InfiniBand stack has to come up inside the initramfs before the node can reach the Warewulf server. PXE succeeds because the HCA firmware handles the early network boot. The failure happens later, when the Linux kernel takes over without the required InfiniBand drivers in the initramfs.

Problem

Nodes configured for two-stage dracut boot over InfiniBand fail to download the system and image stages from the Warewulf server. The node gets past iPXE (kernel and initramfs download successfully) but the network is not available when the wwinit dracut module tries to fetch stages via curl.

Symptoms

The node console shows NetworkManager timing out (up to 300 seconds) followed by:

dracut-pre-mount: gzip: stdin: unexpected end of file
dracut-pre-mount: cpio: premature end of archive
dracut: FATAL: Unable to load stage: system

The warewulfd logs on the server show kernel and initramfs stage requests from the node, but no system or image stage requests.

If you drop into the dracut shell (rd.shell kernel argument) and run ip a, the InfiniBand interface will be present but have no IP address.

Resolution

Install rdma-core in the image

The rdma-core package provides the rdma dracut module, which tells dracut to include InfiniBand kernel modules and RDMA configuration in the initramfs.

wwctl image exec --build=false <image> -- /usr/bin/dnf -y install rdma-core

Rebuild the initramfs with the rdma module

Include both wwinit and rdma in the dracut add list:

wwctl image exec <image> -- /usr/bin/dracut --force --no-hostonly --add "wwinit rdma" --regenerate-all

Verify the initramfs contains IB modules

wwctl image exec --build=false <image> -- /usr/bin/bash -c 'lsinitrd /boot/initramfs-*.img | grep -E "mlx5_ib|ib_ipoib"'

You should see mlx5_ib.ko.xz and ib_ipoib.ko.xz in the output. If these are missing, the IB interface will not come up during boot.

Root Cause

Without rdma-core, dracut does not include InfiniBand kernel modules (mlx5_ib, ib_ipoib, ib_core) in the initramfs. The kernel-network-modules dracut module only covers drivers/net/, which includes Mellanox Ethernet drivers but not the InfiniBand modules under drivers/infiniband/. The rdma dracut module, provided by rdma-core, is what tells dracut to include the full IB stack.

The iPXE stage works because the HCA's PXE firmware handles networking independently of the Linux kernel.

Notes

• Include the kernel arguments required by your hardware and interface naming. Common examples are rd.driver.pre=mlx5_ib, rd.driver.pre=ib_ipoib, ip=ib0:dhcp, and bootdev=ib0.

References & related articles

Enabling Two-Stage Booting with Dracut on an Existing Warewulf Image Fixing dbus-broker Failures During Two-Stage Dracut Boot Warewulf GitHub Issue #1620 - dracut boot with infiniband only networking