Troubleshooting Warewulf Client Timeout Issues

Stephen Simpson
Senior Customer Support Engineer

Mar 18, 2025

Introduction

In a typical Warewulf environment, compute nodes run wwclient to pull runtime overlays from the head node. A node may fail to connect to the provisioning server after boot, resulting in a client timeout. This article details a step-by-step troubleshooting process to verify connectivity, inspect configuration settings, and adjust the Warewulf client configuration if needed.

Problem

When there is a connection issue, you might observe errors such as:

Job for wwclient.service failed because a timeout was exceeded

Failed to start Warewulf node runtime overlay update

Resolution

Verify connectivity and server configuration

A timeout error generally indicates that the compute node is unable to reach the provisioning server. Specifically, it is not able to download the runtime file over the designated Warewulf port. By default, this port is typically set to 9873. Follow these steps to verify connectivity:

Check the client configuration

Confirm the IP address and port that the wwclient is set to use. For example, review the configuration file and verify the settings from a booted compute node:

cat /warewulf/warewulf.conf | grep -E "ipaddr:|port:"

Expected output (with your server’s IP address and port):

ipaddr: <SERVER_IP>
port: 9873

Test server connection

Use curl to request the provision directory and verify that the server is reachable:

curl -I http://<SERVER_IP>:9873/provision/

A successful response should look like:

HTTP/1.1 200 OK
Content-Length: 3779
Content-Type: text
Date: [Date and Time]

If you receive an error such as curl: (7) Failed to connect to <SERVER_IP>, then it is likely that a firewall rule on the server, or between the node and server, is blocking access. To resolve firewall issues on a system using firewalld, run:

firewall-cmd --permanent --add-service=warewulf
firewall-cmd --reload

Test the runtime file URL

The wwclient downloads a runtime file using a URL with several parameters. The URL format is:

http://{ipaddr}:{port}/provision/{wwid}?assetkey={tag}&uuid={localUUID}&stage=runtime&compress=gz

WWID: Usually the node’s MAC address, which you can verify from the boot command line:
```
cat /proc/cmdline | grep -oP 'wwid=[^ ]+'
```
Asset Key and UUID: Values are from dmidecode. If dmidecode is not available or the values are empty, the key and uuid will default to "Unknown."

Test the download with the url you generate. For example:

curl -I "http://192.168.1.2:9873/provision/aa:bb:cc:dd:ee:ff?assetkey=Unknown&uuid=Unknown&stage=runtime&compress=gz"

A correct response will indicate an HTTP 200 OK status and appropriate content headers. If you can download the file or the headers indicate success, the connection is established correctly.

Create a custom warewulf configuration for the wwclient

If the provisioning network is unavailable after boot and you need a custom configuration with a different IP address, you can create a new overlay for the wwclient. By default, the client reads its configuration from /warewulf/warewulf.conf, which is copied from /etc/warewulf/warewulf.conf on the host server. An issue has been submitted for the ip address to be configured at the host or profile level. Until then, and for Warewulf 4.6.0 and below, proceed with the following steps to create a customized overlay:

Create a new overlay

Create an overlay called custom-wwclient:

wwctl overlay create custom-wwclient

Import and edit the Warewulf configuration file

Copy the existing configuration file into the new overlay and open it for editing. For example, to copy it to a custom location such as /warewulf/custom-warewulf.conf:

wwctl overlay import -p custom-wwclient /etc/warewulf/warewulf.conf /warewulf/custom-warewulf.conf
wwctl overlay edit custom-wwclient /warewulf/custom-warewulf.conf

In your editor, modify the ipaddr field to the appropriate IP Address. You can also change the port here if desired.

Modify the systemd unit file

Next, adjust the systemd unit file so that the wwclient uses the new configuration file. Edit the wwclient service as follows:

wwctl overlay edit -p custom-wwclient /etc/systemd/system/wwclient.service

Update or add the following:

[Unit]
Description=Warewulf Node Runtime Overlay Update
After=network.target local-fs.target

[Service]
Type=notify
ExecStart=/warewulf/wwclient --warewulfconf /warewulf/custom-warewulf.conf
ExecReload=/bin/kill -s SIGHUP "$MAINPID"
PIDFile=/var/run/wwclient.pid
TimeoutSec=60

[Install]
WantedBy=multi-user.target

Link the overlay with a profile and build

Add the new overlay to the appropriate profile, in our case default, ensuring to keep any existing overlays. We will be adding this as a system overlay so it is applied at boot.

wwctl profile set default --wwinit=wwinit,custom-wwclient

Then, build the new overlay settings:

wwctl overlay build

Reboot and verify

Reboot the node to ensure it picks up the new configuration. Once up, verify that the process is running with the adjusted configuration file:

ps aux | grep wwclient

You should see a command line similar to:

/warewulf/wwclient --warewulfconf /warewulf/custom-warewulf.conf

References & Related Articles

Rocky Linux Firewalld Guide
Warewulf Documentation
Warewulf Overlays