Ansible Tasks Killed by systemd Idle Session Timeout on Hardened Hosts

Introduction

On security-hardened hosts, a long-running Ansible task can fail partway through with the target host suddenly becoming unreachable, even though the task itself is working correctly. This is common on systems that apply a hardening profile such as DISA STIG or CIS, where systemd is configured to terminate idle login sessions after a fixed period. This article explains why it happens and how to structure your playbook so that long tasks survive the timeout.

This applies to any Ansible run over SSH against a hardened Enterprise Linux host. It is not specific to any single package, task, or Ansible version. It is especially relevant to Rocky Linux from CIQ (RLC) Pro Hardened, where applying the STIG-aligned hardening controls is an available option that can enable this session timeout.

Problem

Ansible connects to managed hosts over SSH and, for the duration of a task, waits for the remote module to return. When a task takes a long time (large package transactions, image builds, bulk file copies, database imports), the SSH session sits and waits with no interactive traffic.

On a hardened host, systemd-logind may be configured to close sessions it considers idle. When it does, it tears down the login session that Ansible is running under, the SSH connection drops, and Ansible reports the host as unreachable. The task was progressing normally right up to the moment the session was killed, which makes the failure look intermittent and unrelated to the work being done. On RLC Pro Hardened this setting comes from the STIG-aligned hardening controls when they are applied, so the behavior appears once that hardening is in place rather than on a default install.

The relevant setting lives in /etc/systemd/logind.conf (or a drop-in under /etc/systemd/logind.conf.d/):

[Login]
StopIdleSessionSec=900

A value of 900 terminates sessions after 15 minutes of perceived inactivity. Any single task that runs longer than this window is at risk.

Notes and Warnings

ℹ️ NOTE This is enforced by the operating system, not by SSH. Client-side SSH keepalives (ServerAliveInterval, ClientAliveInterval) and connection reuse settings (ControlMaster, ControlPersist) do not prevent it, because systemd-logind is closing the session regardless of SSH-level traffic.

Symptoms

A task fails with an unreachable error while other hosts in the same play, running the identical task, complete without issue:

TASK [packages : Apply pending updates] ****************************************
changed: [node001]
changed: [node002]
fatal: [node003]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Shared connection to 10.0.0.30 closed.", "unreachable": true}

The failure tends to correlate with the hosts that have the most work to do (the largest transaction, the biggest copy) rather than with any particular host or package. Re-running the play often succeeds, because the second run has less work left and finishes inside the timeout window.

You can confirm the timeout is in effect on a target host:

grep -ri StopIdleSessionSec /etc/systemd/logind.conf /etc/systemd/logind.conf.d/ 2>/dev/null

Resolution

The goal is to run the long operation in a way that does not depend on the login session staying open. The most reliable approach is to launch the work in a transient systemd service unit, which runs under systemd (PID 1) rather than under your SSH login session, and then poll that unit for completion so the playbook still tracks the result.

Instead of running a long command directly in a task:

- name: Apply pending updates
  ansible.builtin.dnf:
    name: "*"
    state: latest

Launch it in a named transient service unit, then wait for that unit to finish and check how it ended:

- name: Launch pending updates in a detached transient unit
  ansible.builtin.command:
    cmd: systemd-run --unit=ansible-os-update dnf -y upgrade
  changed_when: true

- name: Wait for the update unit to finish
  ansible.builtin.command:
    cmd: systemctl is-active ansible-os-update
  register: update_unit
  until: update_unit.stdout not in ["active", "activating"]
  retries: 60
  delay: 60
  changed_when: false
  failed_when: false

- name: Fail if the update unit did not complete cleanly
  ansible.builtin.fail:
    msg: "The update unit ended in state '{{ update_unit.stdout }}'"
  when: update_unit.stdout == "failed"

Running systemd-run without --scope starts the command as a transient .service unit owned by systemd (PID 1), so it is no longer part of the SSH login session. If systemd-logind closes the session, the unit keeps running to completion. systemd-run returns as soon as the unit has started, so the play does not hold the connection open for the whole job. The wait task polls systemctl is-active until the unit is no longer running, and the final task fails the play if the unit ended in the failed state. A failed transient unit is retained by systemd until it is reset, so that state is always available to check.

Avoid systemd-run --scope for this. A scope unit is executed by systemd-run as its own child and inherits the caller's environment, so it stays inside the login session and can be torn down along with it. The default service mode is what actually detaches the work.

This pattern generalizes to any long operation, not just package updates. For example, a bulk copy or extract launches the same way, with its own unit name:

- name: Extract large dataset in a detached transient unit
  ansible.builtin.command:
    cmd: systemd-run --unit=ansible-dataset-extract tar -xzf /tmp/dataset.tar.gz -C /srv/data
  changed_when: true

You then wait on ansible-dataset-extract with the same systemctl is-active check shown above.

Because the unit is detached from the session, you do not have to wait on it right away. If a job's runtime is unpredictable, or you want to start it on many hosts and reconcile later, launch the unit, do other work, and poll for completion further down the play:

- name: Start long migration in a detached transient unit
  ansible.builtin.command:
    cmd: systemd-run --unit=ansible-migration /opt/scripts/migrate.sh
  changed_when: true

- name: Do other work while the migration runs
  ansible.builtin.debug:
    msg: "Migration started, continuing with other tasks"

- name: Wait for the migration to finish
  ansible.builtin.command:
    cmd: systemctl is-active ansible-migration
  register: migration_unit
  until: migration_unit.stdout not in ["active", "activating"]
  retries: 180
  delay: 60
  changed_when: false
  failed_when: false

Because the work lives in a systemd unit rather than an Ansible async job, an SSH session that drops between starting the unit and polling it does not affect the running job. The unit keeps going, and the next poll picks up its state.

Alternative approaches

There are other ways to address this, with different trade-offs:

Raise or disable the idle timeout on the managed hosts. You can set a larger StopIdleSessionSec (or 0 to disable it) via a drop-in file. This is the simplest change, but on a STIG- or CIS-hardened fleet the timeout is usually mandated by a compliance control, so relaxing it may put the host out of compliance. Prefer a drop-in under /etc/systemd/logind.conf.d/ over editing logind.conf directly, since the main file is package-owned and can be overwritten on update.

⚠️ WARNING Changing StopIdleSessionSec on a hardened host alters a security control. Confirm with whoever owns the hardening baseline before doing this, and apply it through a drop-in file rather than editing the package-owned logind.conf.

Split the work into smaller tasks. Breaking one long transaction into several shorter ones can keep each task inside the timeout window. This avoids touching the security configuration, but it is fragile: as the amount of work grows over time, individual tasks can drift back over the limit, and not every operation can be cleanly subdivided.

The transient service unit approach (systemd-run without --scope) is the recommended option because it survives the timeout without modifying the security baseline and works for any long-running command.

Root Cause

Ansible holds the SSH login session open while it waits for a module to return. A hardened host running systemd-logind with StopIdleSessionSec set treats that waiting session as idle and terminates it once the configured interval elapses. The task is not failing on its own merits; the session it depends on is being closed out from under it. Detaching the work into a transient systemd service unit removes that dependency, so the operation completes regardless of what happens to the login session.

References & related articles

systemd-run manual (Rocky Linux 9)
Ansible asynchronous actions and polling