Speeding Up Ascender Jobs with SSH Pipelining

Introduction

SSH pipelining reduces the runtime of Ascender job templates that run many short tasks. Enabling it takes one configuration change, with no additional software on the managed hosts and no execution environment rebuild. In the benchmark below it cut runtime by a third on a local network, and the benefit grows with network latency.

Set ANSIBLE_PIPELINING to true under Settings, then Jobs, then Extra Environment Variables. Then use verbose job output to confirm pipelining is active.

Ascender runs playbooks through the standard Ansible ssh connection plugin, so this applies to every job template that reaches a host over SSH. The same settings work in a plain ansible-playbook run outside Ascender.

The overhead is paid per task and per host, so the total saving scales with the size of the inventory.

Problem

By default, Ansible needs several SSH operations to run one module. It resolves the remote temporary directory path, creates the directory, transfers the module payload with sftp, adjusts permissions, executes the module, then removes the directory. For a plain module such as command or stat, with no privilege escalation or escalation to root, that is six operations at the SSH layer.

The exact count depends on the module, the privilege escalation settings, and the Ansible version. Each operation adds at least one network round trip, a local SSH client process, and a remote channel setup. For modules that finish in milliseconds, that overhead can dominate the runtime of a playbook containing hundreds of small tasks.

With pipelining enabled, Ansible sends the module payload to the remote Python interpreter through standard input. That avoids the temporary directory, the file transfer, and the cleanup, reducing six SSH operations to one.

Pipelining does not reduce the number of SSH connections. Ansible already reuses one connection per host by default, through the ControlMaster and ControlPersist options in its default ssh_args. Pipelining reduces the number of operations carried through that connection. The benchmark below measures connection reuse and pipelining separately.

Resolution

Enable pipelining

Pipelining is a connection setting and does not require an execution environment rebuild. Enable it at the instance, project, or inventory level.

To enable pipelining across Ascender, add the following job environment variable under Settings, then Jobs, then Extra Environment Variables. It applies to every playbook run on the instance, regardless of job template or execution environment:

{"ANSIBLE_PIPELINING": "true"}

For a single project, set it in the project's ansible.cfg. Ascender runs playbooks with the project directory as the working directory, so this file is honored:

[ssh_connection]
pipelining = True

To enable pipelining for selected hosts, set the connection variable in the Variables field of an Ascender inventory, group, or host. This supports a staged rollout and comparison against hosts that do not have it enabled. You can also set the variable in play vars:

ansible_pipelining: true

To confirm pipelining is active, set the job template's Verbosity field to 3 (Debug) and relaunch. A pipelined module task shows no PUT or sftp lines in the job output.

Expected gain

Measured against one managed host on a local network, 45 module executions per pass, changing only the two connection settings:

Connection reuse	Pipelining	Runtime
Off	Off	91.0 s
On (Ansible default)	Off	31.3 s
Off	On	31.2 s
On (Ansible default)	On	21.0 s

Against a stock configuration, which already has connection reuse enabled, pipelining cut runtime by 33 percent. The gain increases with network latency to the managed hosts.

Notes

Not every task is pipelined

Pipelining applies only to standard Python modules. Action plugins that transfer files, including copy, template, unarchive, assemble, and script, continue to create a remote temporary directory and transfer their content whether pipelining is enabled or not. A playbook dominated by copy and template tasks will see little improvement. Long-running tasks such as large package transactions or image builds also gain little, because SSH overhead is a small part of their runtime.

Tasks using async also bypass pipelining, as do modules that are not standard Python modules. Pipelining is disabled outright when ANSIBLE_KEEP_REMOTE_FILES is set, and the su become plugin does not support it at all.

The requiretty caveat

Historically, requiretty prevented pipelining from working with privilege escalation. Without pipelining, Ansible passes -tt to allocate a pseudo-terminal, which satisfies the requiretty setting in sudoers. With pipelining, the module payload occupies standard input, so Ansible cannot use -tt. Under Defaults requiretty, sudo then fails with an error indicating that a terminal is required. That compatibility problem is why pipelining ships disabled.

Current Enterprise Linux releases do not enable requiretty by default. It is off by default in upstream sudo and is not set in the shipped sudoers file on Rocky Linux 8, 9, or 10. RHEL removed it from the default configuration in version 7.3. Hosts upgraded in place from older releases may retain the setting, so check before enabling pipelining across the fleet:

sudo grep -r requiretty /etc/sudoers /etc/sudoers.d/ 2>/dev/null

requiretty and use_pty are separate settings. Hardening profiles such as CIS and STIG mandate Defaults use_pty, and sudo 1.9.14 and later enable it by default. When sudo is not attached to a terminal, as with pipelining, use_pty has no effect and does not interfere with pipelining.

Keep the default ControlPersist setting

A dropped multiplexed connection can cause a broken pipe error, and Ansible may not retry the operation successfully. Keep ControlPersist at its default of 60 seconds.

References & related articles

Ansible ssh connection plugin documentation
Ansible configuration reference
Ascender job templates, including the Verbosity field
Ascender inventories, groups, and host variables
Ansible Tasks Killed by systemd Idle Session Timeout on Hardened Hosts