How to Copy Files to Warewulf Nodes

Introduction

This article addresses how to distribute files to configured and active compute nodes within a Warewulf managed cluster. It outlines three different methods: using Warewulf's native runtime overlay system, leveraging Ansible for configuration management, and using traditional scp or pscp commands.

Problem

The user needs a method to copy files from the Warewulf server to a specific location on all or certain active compute nodes.

Resolution

There are several ways to achieve this. Below are three common approaches:

Option 1: Warewulf Runtime Overlays

Warewulf's overlay system is a powerful way to manage node configurations. Runtime overlays are particularly suited for distributing files or configurations that might need to be updated while nodes are running, as they are periodically pulled by the nodes from the Warewulf server. You can read more information about overlays in Warewulf's documentation. Here is a basic overview:

Follow these steps to distribute a file using a runtime overlay:

1. We start by creating an overlay for our files.

   wwctl overlay create my_file_distribution_overlay
   

2. Next, we will copy the local file into the overlay, specifying its destination path on the compute nodes. The --parents flag ensures that the directory structure is created within the overlay if it doesn't exist. Copy any other files you want into this as well.

   wwctl overlay import --parents my_file_distribution_overlay /path/to/your/local/file.txt /path/on/node/destination_file.txt
   

3. Now we can assign the overlay to a profile's runtime overlays. First, identify the existing runtime overlays for the target profile (e.g., default) to ensure you don't unintentionally remove them.

   wwctl profile list default -a | grep 'RuntimeOverlay'
   

   Let's say the output shows default RuntimeOverlay hosts,ssh.authorized_keys. You would then add your new overlay (my_file_distribution_overlay) to this list.

   wwctl profile set default -R hosts,ssh.authorized_keys,my_file_distribution_overlay
   

   Note: The wwctl profile set -R command replaces the entire list of runtime overlays for the specified profile with the new list you provide. Ensure you include all desired runtime overlays (both existing and new) in the command.

4. Finally, we can build the overlay to make it available to the nodes.

   wwctl overlay build
   

Once built and assigned, nodes configured with this profile will fetch and apply the runtime overlay, including your distributed file, during their next update cycle which usually runs every 5 minutes.

Option 2: Using Ansible

If you are already familiar with and use Ansible in your environment, it can be a convenient way to distribute files.

1. Save the following content as a playbook YAML file (e.g., distribute_file.yml):

   ---
   - name: Distribute a file to Warewulf compute nodes
     hosts: warewulf_compute_nodes # Ensure this group is defined in your Ansible inventory
     gather_facts: false
     tasks:
       - name: Copy the specified file to nodes
         ansible.builtin.copy:
           src: /path/to/your/local/file.txt
           dest: /path/on/node/destination_file.txt
           owner: root  # Optional: set file owner
           group: root  # Optional: set file group
           mode: '0644' # Optional: set file permissions
   

2. Run the playbook:

   ansible-playbook -i /path/to/your/inventory_file distribute_file.yml
   

This method is particularly useful if you need to perform other configuration tasks in conjunction with file distribution or if Ansible is already integrated into your workflow. For a complete automation solution, check out Ascender Ascender is a fully open source alternative to Ansible Automation Platform or Ansible AWX that automates tasks like provisioning, configuration management, application deployment, and orchestration of your infrastructure and network processes.

Option 3: Using SCP or PSCP (Parallel SCP)

SCP

The simplest method to copy a file over to your nodes would be to loop through a set of hosts and scp the file to your server. First, we need to create a hosts file (e.g., hosts.txt) containing the hostnames or IP addresses of your target compute nodes, one per line. By default, Warewulf will add the node names to your /etc/hosts, allowing you to reference them without needing to rely on the IP address. As an example:

# Example hosts.txt:
node01
node02
node03

Then, we can use a shell loop to scp the file:

for host in $(cat hosts.txt); do
  scp "/path/to/your/local/file.txt" "root@${host}:/path/on/node/destination_file.txt"
done

PSCP

If you have a lot of servers that you need a file copied to, you can take advantage of pscp from the pssh package. This will run the scp process in parallel. Using the same hosts.txt file:

pscp.pssh -l "root" -h hosts.txt "/path/to/your/local/file.txt" "/path/on/node/destination_file.txt"

You can also use the wwctl command to pull nodes and filter based on certain criteria with jq. jq is a lightweight and flexible command-line tool for processing and querying JSON data. We will use wwctl node list -j and jq to dynamically target nodes. In this example, we will target all nodes in the default profile.

pscp.pssh -l "root" -h <(wwctl node list -j | jq -r 'to_entries[] | select(.value.profiles | index("default")) | .key') "/path/to/your/local/file.txt" "/path/on/node/destination_file.txt"

Notes

• To install jq in Rocky Linux, you can run dnf install jq, which will come from the baseos repository.

References

Warewulf Overlay Guide
Ansible Copy Documentation
Ascender Automation