A profile.d Script Printing to stdout Breaks rsync and SSH on Warewulf Nodes

Introduction

When you copy files to Warewulf nodes with rsync or scp, or run remote commands over ssh (including wwctl ssh), each of these opens a non-interactive shell on the node, and that shell still runs its startup files. Any startup code that writes to standard output during the session, commonly a script in /etc/profile.d/, injects its output into the session and can break it. This article shows how to recognize and fix that class of failure, using a Slurm bash-completion script as the worked example. Because it is a shell and SSH behavior rather than a Warewulf-specific one, it applies to Rocky Linux 8, 9, and 10 images on Warewulf 4.6 and later.

Problem

rsync, scp, or wwctl ssh to a node fails. Instead of transferring, the command returns text that clearly came from a login script rather than from the transfer. For example, a Slurm bash-completion script in /etc/profile.d/ prints its dependency message when sourced in a non-interactive shell:

$ rsync example.log node002:~/
FATAL: Missing required source file.
 (1) Install package 'bash-completion'
 (2) $ source /usr/share/bash-completion/bash_completion
 (3) $ source slurm_completion.sh

The specific text depends on which script is printing; the pattern is the same. Some login script in /etc/profile.d/ writes to standard output during a non-interactive SSH session, and that output corrupts the rsync stream and aborts the transfer. (Here the message is also misleading: bash-completion is already installed. The completion helper functions are simply not loaded in a non-interactive shell, which is what makes the script print its guidance.)

Symptoms

• rsync, scp, or wwctl ssh to a node returns login-script text instead of completing.

• Running a non-interactive command over SSH prints something when it should be silent:

  ssh node002 true
  

  Any output from that command is the offending script. A clean node prints nothing.

• The failure is often node-dependent, because the triggering script is present or active on only some nodes.

Resolution

Find the script in /etc/profile.d/ that produces output, then either move it (if it is a shell-completion script) or stop it from printing in non-interactive shells.

Find the offending script

ssh <node> true prints the offending output, which usually names the script or is recognizable. If it is not obvious, inspect the scripts that /etc/profile.d/ sources:

ssh node002 true
ls -l /etc/profile.d/

Fix it

If the culprit is a shell-completion script (as with Slurm's slurm_completion.sh), move it out of /etc/profile.d/ into a bash-completion directory. Those directories are sourced on demand by the bash-completion framework only in interactive shells, so the script never runs during a transfer, and tab completion still works:

sudo mv /etc/profile.d/slurm_completion.sh /etc/bash_completion.d/

/usr/share/bash-completion/completions/ works equally well if you prefer the system completions location.

If the script genuinely needs to stay in /etc/profile.d/ (for example it sets environment variables and only prints under some condition), guard its output so it runs only in an interactive shell:

case $- in
  *i*) : # interactive-only output goes here
       ;;
esac

Confirm the transfer now succeeds and the non-interactive check is clean:

ssh node002 true          # no output
rsync example.log node002:~/ # transfers with no FATAL block

Make the fix persistent

A node's /etc/profile.d/ scripts come from the Warewulf image or an overlay, so a live edit is undone on the next reboot or reprovision. Apply the same change at the source and rebuild:

• If the script is baked into the node image, fix it in the image, rebuild the image, and reboot the affected nodes.
• If it is delivered by an overlay, make the change in that overlay and run wwctl overlay build. Note that overlays cannot delete a file the image already provides, so if the script is in the image, fix it in the image.

Root Cause

The remote end of an rsync or scp transfer (and ssh <host> <command>) runs a non-interactive shell that still reads its shell startup files. When startup code writes to standard output, rsync reads the unexpected bytes on the remote side as protocol data, cannot parse them, and aborts. On Rocky Linux, scripts in /etc/profile.d/ are a common source of such output, since they are sourced during shell startup. For example, a Slurm completion script (slurm_completion.sh) in /etc/profile.d/ prints its dependency instructions when sourced outside an interactive shell. Relocating a completion script to a bash-completion directory, or guarding a script's output behind an interactive-shell check, keeps the non-interactive session clean while preserving the script's purpose.

Notes

• Nothing in /etc/profile.d/ should write to standard output or standard error unconditionally. It is not specific to Slurm: MOTD-style banners, conda/toolchain init blocks, and other completion scripts cause the identical failure. The rule is the same for any of them.
• Warewulf makes this a cluster-wide problem rather than a one-off, because the image and overlays deliver the same /etc/profile.d/ scripts to every node. A script that prints will break transfers on every node that carries it.
• The reverse is also worth remembering when writing provisioning scripts: never rely on /etc/profile.d/ for output, and keep it limited to setting environment for both interactive and non-interactive shells.

References & related articles

Distributing Slurm Configuration to Compute Nodes Using Configless Mode Warewulf Documentation - Overlays