Setting Up Rsyslog with Encrypted Transport on Rocky Linux
Introduction
This guide focuses on troubleshooting issues encountered during or after setting up rsyslog for encrypted log transport on Rocky Linux. It addresses common problems such as missing modules and misconfigured parameters that can prevent proper decryption of incoming logs. Whether you’re building a centralized logging infrastructure or enhancing your lab environment’s security, this guide provides hands-on insights into troubleshooting and correct configuration.
Problem
When an rsyslog
collector is configured to receive logs from multiple systems and intended to secure the log transport using encryption, errors indicating required modules are missing or misconfigured can be encountered. This can cause scenarios where logs are received on the collector, but are not being decrypted correctly, halting proper secure communication.
Symptoms
The rsyslog
configuration can produce several notable errors:
Module loading issue
could not load module 'gtls', errors: trying to load module /usr/lib64/rsyslog/gtls.so: /usr/lib64/rsyslog/gtls.so: cannot open shared object file: No such file or directory
Parsing errors in the configuration file
error during parsing file /etc/rsyslog.d/01-collector.conf, on or before line 9: parameter 'StreamDriver.AuthMode' not known -- typo in config file?
error during parsing file /etc/rsyslog.d/01-collector.conf, on or before line 9: parameter 'StreamDriver.Mode' not known -- typo in config file? error during parsing file /etc/rsyslog.d/01-collector.conf, on or before line 9: parameter 'StreamDriver.Name' not known -- typo in config file?
Even though logs are being received, they remain encrypted and unprocessed, indicating a configuration or module issue.
Resolution
Below are the prerequisites and detailed steps to resolve the issue:
Testing
-
Utilize the
root
account or a user withsudo
privileges for testing. -
Set up two Rocky Linux 8.10 machines: one designated as the server and the other as the client.
-
Set the hostname for each node with an FQDN:
-
Server:
hostnamectl set-hostname rocky-linux810-server.rsyslog.test
-
Client:
hostnamectl set-hostname rocky-linux810-client.rsyslog.test
-
-
Update the
/etc/hosts
file on both nodes with the respective IP addresses and hostnames. An example is below (please replace the IPs and hostnames with your own):
cat << "EOF" | tee /etc/hosts
10.0.0.4 rocky-linux810-server.rsyslog.test rocky-linux810-server
10.0.0.3 rocky-linux810-client.rsyslog.test rocky-linux810-client
EOF
Package installation
On both the server and client, install the necessary packages using the following command:
dnf install -y rsyslog rsyslog-gnutls rsyslog-openssl logrotate
Firewall and SELinux adjustments
- Open port
6514/tcp
on both nodes:
firewall-cmd --add-port=6514/tcp --permanent
firewall-cmd --reload
- On the server, allow port
6514
through SELinux:
semanage port -m -t syslogd_port_t -p tcp 6514
Certificate generation
- Create a directory for certificates:
mkdir -p /etc/rsyslog/keys
- Generate a self-signed certificate and private key:
openssl req -new -x509 -days 365 -nodes \
-out /etc/rsyslog/keys/server-cert.pem \
-keyout /etc/rsyslog/keys/server-key.pem
Directory setup for log storage
- Create directories for logs on both the server and client:
# Server
mkdir -p /data/logs/rocky-linux810-server
# Client
mkdir -p /data/logs/rocky-linux810-client
- Set appropriate permissions for each directory:
# Server
chmod 644 /data/logs/rocky-linux810-server
# Client
chmod 644 /data/logs/rocky-linux810-client
- Configure SELinux contexts:
# Server
semanage fcontext -a -t var_log_t "/data/logs/rocky-linux810-server(/.*)?"
restorecon -Rv /data/logs/rocky-linux810-server
# Client
semanage fcontext -a -t var_log_t "/data/logs/rocky-linux810-client(/.*)?"
restorecon -Rv /data/logs/rocky-linux810-client
Correcting the module issue
- The key resolution is to ensure the
gtls
module is correctly referenced. Since the file/usr/lib64/rsyslog/gtls.so
module is missing, a symbolic link needs to be created:
ln -s /usr/lib64/rsyslog/lmnsd_gtls.so /usr/lib64/rsyslog/gtls.so
Server configuration (/etc/rsyslog.d/01-collector.conf
)
- Create/update the configuration file with the following content:
cat << "EOF" | tee /etc/rsyslog.d/01-collector.conf
# /etc/rsyslog.d/01-collector.conf
module(load="imtcp")
module(load="gtls")
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/pki/tls/certs/ca-bundle.crt
$DefaultNetstreamDriverCertFile /etc/rsyslog/keys/server-cert.pem
$DefaultNetstreamDriverKeyFile /etc/rsyslog/keys/server-key.pem
$InputTCPServerRun 6514
$template RemoteLogs,"/data/logs/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
& ~
EOF
Client configuration (/etc/rsyslog.d/01-sender.conf
)
- Create/update the configuration file on the client with:
cat << "EOF" | tee /etc/rsyslog.d/01-sender.conf
module(load="omfwd")
module(load="gtls")
$DefaultNetstreamDriverCAFile /etc/pki/tls/certs/ca-bundle.crt
$DefaultNetstreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode anon
*.* action(type="omfwd"
target="10.0.0.4"
port="6514"
protocol="tcp")
EOF
Service restart and testing
- Restart
rsyslog
on both the server and client:
systemctl restart rsyslog
- Test by sending an encrypted message from the client:
logger "THIS IS A TEST ENCRYPTED MESSAGE"
- On the server, verify the decrypted log messages:
tail -f /data/logs/rocky-linux810-client/root.log
- An example output:
tail -f /data/logs/rocky-linux810-client/root.log
[root@rocky-linux810-server ~]# tail -f /data/logs/rocky-linux810-client/root.log
Apr 30 07:26:07 rocky-linux810-client root[53589]: THIS IS A TEST ENCRYPTED MESSAGE
Apr 30 07:26:07 rocky-linux810-client root[53589]: THIS IS A TEST ENCRYPTED MESSAGE
Apr 30 07:26:08 rocky-linux810-client root[53590]: THIS IS A TEST ENCRYPTED MESSAGE
Root Cause
The core issue stems from a missing symbolic link for the gtls
module. The rsyslog
configuration is attempting to load the module from /usr/lib64/rsyslog/gtls.so
, which does not exist. Instead, the system provides the library as /usr/lib64/rsyslog/lmnsd_gtls.so
. Creating the symbolic link resolves the module-not-found
error and allows the proper configuration directives—such as StreamDriver.Name
, StreamDriver.Mode
, and StreamDriver.AuthMode
—to be recognized and processed, thereby enabling successful decryption of incoming logs.
Notes
-
SELinux and Firewall: Always ensure that SELinux contexts and firewall rules are appropriately configured, as they can block the secure transport on non-standard ports like
6514
. -
Module Naming and Configuration: Module names and parameters can vary between rsyslog releases. It’s advisable to consult the official rsyslog documentation when encountering similar issues.
-
Certificate Management: For production environments, replace self-signed certificates with those issued by a trusted CA and ensure proper certificate management policies are in place.
References & Related Articles
Rsyslog Error 2066 - Missing gtls Module
Rsyslog Error 2207 - Unknown Parameter Issues
Rsyslog gtls Network Stream Driver