Unable to SSH to Compute Nodes After Image Update

Introduction

After pushing Warewulf image updates to compute nodes, SSH key-based authentication stops working, prompting for passwords instead. This article addresses the scenario where the Warewulf cluster SSH key is missing from the authorized_keys overlay after an image update.

Problem

SSH access to compute nodes fails with password prompts instead of accepting SSH key authentication. Users who previously had passwordless SSH access to compute nodes can no longer connect using their SSH keys.

Symptoms

When attempting to SSH to a compute node, the system prompts for a password instead of accepting the SSH key:

ssh root@node001
root@node001's password:

The ssh.authorized_keys overlay is present in the node configuration, but SSH key authentication still fails.

Resolution

The issue occurs when the Warewulf cluster SSH key is missing from the head node's authorized_keys file, which is used to populate the compute node overlay. Follow these steps to restore SSH access:

Verify the SSH Overlay Configuration

First, check if the ssh.authorized_keys overlay is configured for the affected node:

wwctl node list <node> -a | grep Overlay

You should see output similar to:

node001 RuntimeOverlay  default  hosts,ssh.authorized_keys,syncuser

The ssh.authorized_keys entry confirms the overlay is configured.

Check the Current Overlay Content

Verify what SSH keys are currently in the overlay for the compute node:

wwctl overlay show -r <node> ssh.authorized_keys /root/.ssh/authorized_keys.ww

This displays the contents of the authorized_keys file that will be deployed to the compute node. If the cluster key is missing, you'll need to add it as shown below.

Verify the Cluster Key Exists

Check that the cluster SSH key pair exists on the head node:

ls -l /root/.ssh/

Look for files named cluster and cluster.pub. These are generated for the cluster during Warewulf installation.

Add the Cluster Key to Authorized Keys

Add the cluster public key to the head node's authorized_keys file:

cat ~/.ssh/cluster.pub >> ~/.ssh/authorized_keys

The ssh.authorized_keys overlay will, by default, use the server's /root/.ssh/authorized_keys file.

Verify the Key Was Added

Confirm the cluster key is now present in the overlay:

wwctl overlay show -r <node> ssh.authorized_keys /root/.ssh/authorized_keys.ww

You should now see the cluster key in the output.

Rebuild the Overlay

Rebuild the overlay to push the changes to the compute nodes:

wwctl overlay build

Since ssh.authorized_keys is typically a runtime overlay, changes should propagate to compute nodes within approximately 60 seconds.

Test SSH Access

After waiting about a minute for the overlay to sync, test SSH access to the compute node:

ssh <node>

SSH key authentication should now work without prompting for a password.

Notes

• The ssh.authorized_keys overlay is typically configured as a runtime overlay, meaning changes propagate automatically without requiring a node reboot.
• The cluster SSH key (~/.ssh/cluster and ~/.ssh/cluster.pub) is used by Warewulf for internal cluster communication.
• Always verify the overlay configuration before and after making changes to ensure the proper keys are being deployed.

References & Related Articles

Warewulf Overlays Documentation
Warewulf Overlays Documentation: SSH Section