Your cmc.log contains errors that read 'Error creating RRD: Too many links' or 'Error during inventory update of host 'myhost': OSError(31, ‘Too many links’)'.
Additionally, you might experience issues with graphing or the HW/SW inventory.

There are two main possible reasons you end up in this situation:

• You are running a Checkmk Appliance installed before July 2021 (Firmware below 1.4.16). Even if you updated to the latest firmware, you might still be affected.

• You are running your own operating system and it either was upgraded from an old ext3 filesystem, or your own filesystem settings impose the limits.

If the first reason does not apply to you, this article cannot provide help. Talk to your Linux administrator to understand how to remediate the situation in your environment.
If you are running a Checkmk Appliance however, read on for a solution.

Before looking at the root of the issue, there is one quick workaround for this issue: Clean up unneeded files.

Double-check your “Automatic disk space cleanup” settings: “Cleanup abandoned host files older than” should always be enabled and set to a sensible timeframe (default is 30 days).
The other settings can be adapted to your needs, but do not have to be changed.

Next, you can search for empty directories, which can be removed without risk of data loss:

If the above command yields output, you can delete the empty directories by running the following command.

After running these commands, a reboot should be performed, to make sure all running processes including the Kernel can create directories again.

Now on to the core of the issue: The limitation of the filesystem. For the Checkmk Appliance the solution is simple, but takes time:

1. Create a full backup of your appliance, including your Checkmk sites.

2. Reset the affected device to factory defaults.

3. Configure the device from scratch to the point, where you can access the formerly created backup.

4. Restore the backup.

While the process takes some time to carry out, it will leave you in a state, where the filesystem is initialized with modern settings. Alternatively, especially in non-clustered environments, where the aforementioned solution means a considerable downtime for your monitoring, you also consider migrating your sites to a new appliance. This can be especially sensible, if you are running a virtual appliance and deploying a new one is comparatively easy.