Distributing Slurm Configuration to Compute Nodes Using Configless Mode
Introduction
When deploying Slurm in a Warewulf-managed cluster, slurm.conf needs to be available on compute nodes. This article covers how to distribute it using Slurm's configless mode, where compute nodes fetch their configuration directly from the controller at runtime. Applies to Warewulf 4.x on Rocky Linux 9.
Problem
With Slurm installed and slurm.conf configured on the controller node, slurm.conf should be distributed to compute nodes and kept in sync whenever it changes.
Resolution
This article assumes Slurm is already installed and slurm.conf is configured on the controller node.
Enable Configless Mode on the Controller
In configless mode, compute nodes fetch slurm.conf directly from the slurmctld controller at runtime, eliminating the need to manage and redeploy configuration files to nodes whenever changes are made.
Add the following parameter to /etc/slurm/slurm.conf on the controller:
SlurmctldParameters=enable_configless
Restart the Slurm controller to apply the change:
sudo systemctl restart slurmctld
Configure Compute Nodes
Warewulf's overlay feature allows distributing configuration files to compute nodes, which is what enables the Slurm integration. The overlay handles placing the necessary slurmd configuration on each node so it can reach the controller and operate in configless mode.
Warewulf Pro includes pre-built Slurm overlays and a node image with Slurm already installed, making it straightforward to get a Slurm-enabled cluster up and running. Install the Slurm overlay package from the CIQ depot on the controller node:
sudo dnf install warewulf-overlays-slurm
After installation, verify the overlay is available:
wwctl overlay list -a | grep -i slurm
Expected output:
slurm etc/ false
slurm etc/munge/ false
slurm etc/munge/munge.key.ww false
slurm etc/slurm/ false
slurm etc/slurm/cgroup.conf.ww false
slurm etc/sysconfig/ false
slurm etc/sysconfig/slurmd.ww false
The overlay includes ready-to-use template files for munge, cgroup, and slurmd configuration, pre-configured for configless mode. Customize the template files for your environment, then assign the overlay to the appropriate nodes or profiles:
sudo wwctl node set node[001-050] --runtime-overlay slurm
sudo wwctl overlay build
If you'd like to know more about Warewulf Pro, contact us.
Notes
- With configless mode, any changes to
slurm.confon the controller are automatically picked up by compute nodes at runtime without requiring overlay rebuilds or node reboots.
References & related articles
Slurm Configless Mode Documentation