/
Troubleshooting Syslog spam with "Failed to acquire watch file descriptor: Permission denied" on newer systemd versions

Troubleshooting Syslog spam with "Failed to acquire watch file descriptor: Permission denied" on newer systemd versions

On newer Linux distributions, periodic Checkmk cron jobs can trigger repeated systemd permission errors, resulting in syslog spam.

LAST TESTED ON CHECKMK 2.4.0P22

Table of Contents

 

Overview

Frequent cron jobs and Checkmk helper scripts can trigger repeated systemd sessions on newer systems, leading to continuous syslog spam with permission errors.


Problem

Systems such as SLES 15 SP7 and RHEL 10.1, and likely also on RHEL 9 and SLES 16, the system journal can become flooded with messages like:

systemd[XXXXX]: Failed to acquire watch file descriptor: Permission denied

These messages typically appear every minute.

 

Example

root@mylinuxhost:~$ sudo journalctl -f | grep -i "permission denied\|Created slice.*{UID Number}" Mar 03 11:00:01 mylinuxhost systemd[1]: Created slice User Slice of UID 466. Mar 03 11:00:01 mylinuxhost systemd[14741]: Failed to acquire watch file descriptor: Permission denied Mar 03 11:00:20 mylinuxhost systemd[1]: Created slice User Slice of UID 466. Mar 03 11:00:20 mylinuxhost systemd[15083]: Failed to acquire watch file descriptor: Permission denied Mar 03 11:01:01 mylinuxhost systemd[1]: Created slice User Slice of UID 466. Mar 03 11:01:01 mylinuxhost systemd[16294]: Failed to acquire watch file descriptor: Permission denied

 

 

Cause

The issue is caused by newer systemd behavior when executing cron jobs for the Checkmk site user.

The following cron jobs are responsible:

# Needed for bulk notifications * * * * * [ ! -e /omd/sites/mysite/etc/check_mk/conf.d/microcore.mk -a -d /omd/sites/mysite/var/check_mk/notify/bulk ] && cmk --notify send-bulks # Cleanup OpenTelemetry collector files * * * * * /omd/sites/mysite/bin/cmk-cleanup-otel-collector-files --older-than 120 "$OMD_ROOT"

Each execution triggers systemd to create a user slice for the site user. Due to missing lingering configuration, systemd cannot properly acquire the required file descriptor, resulting in repeated permission errors.

 

Solution

 You will need to adjust your PAM configuration. This approach stops cron jobs and Checkmk helper scripts from creating a new systemd session every time they run.

By default, both cron and sudo use system-auth, which includes pam_systemd.so. This is what creates a new session and triggers the log messages. By removing this part, the jobs still run normally but no longer create systemd sessions.

Jobs continue to run as expected, authentication and permissions remain unchanged, and only the creation of systemd sessions is disabled.

 

Persistent

The following solution is considered permanent since it will persist after a system reboot. 


Update cron and sudo PAM configurations

  1. First you will need to edit your PAM crond configuration as the root user using an editor.

    root@mylinuxhost:~$ vi /etc/pam.d/crond

     

  2. You will need to comment out session include system-auth like in the example below:

    # The PAM configuration file for the cron daemon # # # Although no PAM authentication is called, auth modules # are used for credential setting auth include system-auth account required pam_access.so account include system-auth session required pam_loginuid.so #session include system-auth

     

  3. Save your changes and restart cron.

    root@mylinuxhost:~$ sudo service cron restart

     

  4. Next, you will need to edit your PAM sudo stack configuration as the root user using an editor.

    root@mylinuxhost:~$ vi /etc/pam.d/sudo

     

  5. You will need to comment out session include system-auth like in the example below:

    #%PAM-1.0 auth include system-auth account include system-auth password include system-auth #session include system-auth session required pam_limits.so

     

  6. Save your changes and restart cron.

    root@mylinuxhost:~$ sudo service cron restart

 

Temporary

The following solution is only a temporary solution since it will reverted once the system reboots. 

 

Systemd lingering

Enable systemd lingering for the Checkmk site user.

This allows the user to run background services without requiring an active login session and prevents the repeated creation of transient user slices.

 

Identify the site user

If you know the UID (example: 466):

root@mylinuxhost:~$ SITE=$(getent passwd {UID Number} | cut -d: -f1) root@mylinuxhost:~$ echo "Site user: $SITE" Site user: mysite

Alternatively, if you already know the site name, you can skip this step.

 

Enable lingering

sudo loginctl enable-linger <site-user>


Example:

sudo loginctl enable-linger mysite

 

Related articles