Error When Symlinking User and Group ID Mapping

Introduction

In environments that use user namespaces, administrators must keep user and group ID mappings consistent across multiple hosts to maintain proper access control. Many try to achieve this by symlinking /etc/subuid and /etc/subgid to shared storage, which distributes the same mappings everywhere.

However, Apptainer and similar tools rely on the newuidmap and newgidmap utilities, which refuse to read symbolic links. When you use symlinks for these files, Apptainer’s fakeroot builds fail. This article explains why this happens and shows safer ways to synchronize mappings across hosts.

Problem

If you symlink /etc/subuid and /etc/subgid to shared storage, Apptainer’s fakeroot feature breaks. The newuidmap and newgidmap utilities reject symlinked files to prevent privilege escalation and ensure file integrity. Because these tools can’t read the mappings, any fakeroot build or container that depends on them fails.

This incompatibility prevents Apptainer from working correctly in environments that distribute user mappings through symbolic links.

Symptoms

When you build an Apptainer image in fakeroot mode on a system with symlinked subuid or subgid files, you might see errors like:

apptainer build --fakeroot rockylinux_fakeroot.sif docker://rockylinux:9
ERROR  : 'newgidmap' execution failed. Check that 'newgidmap' is setuid root or has setcap cap_setgid+eip.
ERROR  : Error while waiting event for user namespace mappings: no event received

These messages show that Apptainer can't create the required user namespace mappings because newuidmap and newgidmap fail to open the symlinked configuration files.

Root cause

newuidmap and newgidmap open the /etc/subuid and /etc/subgid files with the O_NOFOLLOW flag. This flag tells the kernel not to follow symbolic links. When the path ends in a symlink, the system call fails with an ELOOP error "Too many levels of symbolic links".

For example, an strace trace shows the failure:

openat(AT_FDCWD, "/etc/subgid", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW) = -1 ELOOP (Too many levels of symbolic links)

Because of this behavior, the utilities can't access symlinked mapping files, and Apptainer fails to set up user namespaces when using fakeroot mode.

Resolution

To synchronize subuid and subgid files across hosts without using symlinks, consider alternative methods such as:

Rsync and Cron jobs

Use rsync in conjunction with a cron job to periodically sync these files across hosts. For details, refer to the Rocky Linux Rsync Guide and Rocky Linux Cron Guide.

Ascender/Ansible automation

Automate file synchronization with Ansible for a more flexible and scalable solution. Ascender, an orchestration tool from CIQ, can assist in managing these automation tasks. For more details, refer to the Rocky Linux Ansible Guide and Ascender Automation.

References & related articles

Rocky Linux Rsync Guide
Rocky Linux Cron Guide
Rocky Linux Ansible Guide
Ascender Pro