Kernel Panics Caused by CrowdStrike Falcon Sensor on Rocky Linux

Introduction

Production Rocky Linux systems running CrowdStrike Falcon Sensor may experience kernel panics with general protection faults during container operations. The crashes occur in __kmem_cache_alloc_node due to memory corruption originating from the falcon_lsm_serviceable kernel module. This article addresses the specific crash signature, diagnostic indicators, and resolution steps for Falcon-induced kernel panics on Rocky Linux 9.

Problem

Servers running CrowdStrike Falcon Sensor experience kernel panics during container runtime operations, particularly when executing binaries via runc or similar container tools. The crash occurs in the kernel's memory allocator (__kmem_cache_alloc_node) when Falcon's security hooks intercept system calls. Crash analysis shows the fault originates in Falcon's kernel modules.

Symptoms

The crash signature is highly specific and identifiable in crash dumps and dmesg logs:

Primary indicators:

• General protection fault with non-canonical memory addresses such as 0x289d17010b1fa10a or 0x60c5387b1ee398de
• Crash occurs in __kmem_cache_alloc_node+0x10e/0x2e0 or __kmalloc
• Process triggering crash: runc, containerd, or other container execution tools
• Kernel tainted with P (proprietary) and E (unsigned module) flags
• Multiple stack frames from Falcon modules: falcon_lsm_serviceable, falcon_lsm_pinned_*
• Specific Falcon functions in call stack:
  • cshook_systemcalltable_pre_compat_sys_ioctl
  • cshook_security_bprm_check_security
  • pinnedhook_security_bprm_check_security
  • _ZdlPvmSt11align_val_t (C++ memory management in kernel space)

Example crash signature from dmesg:

general protection fault, probably for non-canonical address 0x289d17010b1fa10a: 0000 [#1] PREEMPT SMP NOPTI
CPU: 9 PID: 3903395 Comm: runc Kdump: loaded Tainted: P E ------- --- 5.14.0-503.33.1.el9_5.x86_64 #1
RIP: 0010:__kmem_cache_alloc_node+0x10e/0x2e0

Resolution

Step 1: Confirm Falcon as root cause

Extract and examine the crash dump or dmesg output. Look for the specific Falcon crash signature in the call stack:

Call Trace:
 __kmem_cache_alloc_node+0x10e/0x2e0
 __kmalloc+0x4b/0x140
 cshook_systemcalltable_pre_compat_sys_ioctl+0x3b505/0x43340 [falcon_lsm_serviceable]
 cshook_security_bprm_check_security+0x57/0x70 [falcon_lsm_serviceable]
 pinnedhook_security_bprm_check_security+0x39/0x80 [falcon_lsm_pinned_18108]
 security_bprm_check+0x7b/0x100
 do_execveat_common.isra.0+0x1a4/0x280
 __x64_sys_execve+0x3f/0x50
 do_syscall_64+0x5c/0x80
 entry_SYSCALL_64_after_hwframe+0x49/0xbe

Key diagnostic observation: in typical crashes, Falcon modules account for a significant portion of the call stack such as 23 out of 50 frames, while hardware components like MegaSAS and bnxt_en show zero appearances in the execution path and no errors in dmesg.

To examine crash information from kdump:

# Check the auto-generated dmesg file
grep -A 30 "Call Trace" /var/crash/*/vmcore-dmesg.txt | grep falcon

Step 2: Rule out hardware components

Verify that hardware components aren't contributing to the crashes:

# Verify the RAID controller shows no errors
sudo dmesg | grep -i "mega\|raid" | grep -i error

# Verify the network driver shows no errors
sudo dmesg | grep -i "bnxt\|network" | grep -i error

# Check if these components appear in the crash stack
grep -E "megaraid|bnxt_en|mpt3sas" vmcore-dmesg.txt

If the hardware components are clean and absent from the crash stack, the issue is isolated to Falcon.

Step 3: Contact CrowdStrike support

Engage CrowdStrike support with the crash analysis showing Falcon module involvement:

• Provide the vmcore dump or extracted dmesg logs showing the call stack
• Include the Falcon Sensor version: sudo /opt/CrowdStrike/falconctl -g --version
• Include the kernel version: uname -r
• Reference the specific functions in the crash: cshook_systemcalltable_pre_compat_sys_ioctl, falcon_lsm_serviceable
• Request information about known issues with your kernel version and container runtimes

Step 4: Temporarily disable Falcon for isolation test

On a non-production or isolated affected system, turn off Falcon to definitively confirm if that is the root cause or not:

# Stop Falcon sensor (this should unload modules automatically)
sudo systemctl stop falcon-sensor

# Disable automatic startup
sudo systemctl disable falcon-sensor

# Verify Falcon modules are unloaded
lsmod | grep falcon

# If modules remain loaded, unload them manually
for module in $(lsmod | grep falcon | awk '{print $1}'); do
    sudo modprobe -r "$module"
done

Monitor the system under normal container workload. If kernel panics cease without Falcon loaded, this definitively confirms Falcon as the root cause.

Step 5: Apply resolution from CrowdStrike

Follow the resolution provided by CrowdStrike support. This may include Sensor updates, configuration changes, or kernel version recommendations based on their compatibility matrix.

Step 6: Re-enable and monitor

After applying updates, re-enable Falcon and monitor for any recurrence:

sudo systemctl enable falcon-sensor
sudo systemctl start falcon-sensor

# Verify Falcon modules loaded
lsmod | grep falcon

# Monitor for crashes
sudo journalctl -k -f | grep -i "general protection\|kernel panic"

Notes

• This specific crash pattern occurs when Falcon sensor intercepts container execution (execve system calls) during binary launches.
• The crash occurs when Falcon's cshook_systemcalltable_pre_compat_sys_ioctl function attempts memory allocation and encounters corrupted pointers.
• Diagnostic evidence includes stack traces showing Falcon module involvement and absence of hardware errors in dmesg logs.
• The non-canonical memory addresses indicate pointer corruption within the Falcon module code path.
• This issue affects containerized workloads more frequently than traditional process execution due to the higher rate of execve calls in container environments.
• Falcon sensor versions and kernel compatibility should be verified together - not all sensor versions are certified for all kernel versions.

References & Related Articles

Rocky Linux: How to Enable and Configure kdump Rocky Linux: Kernel Crash Dump Analysis