Troubleshooting rising memory or CPU usage

Troubleshooting rising memory or CPU usage

This article explains how to troubleshoot a Checkmk process with rising memory or CPU usage and collect the information needed for further analysis.

LAST TESTED ON CHECKMK 2.4.0P35

Overview

This article explains how to investigate a Checkmk process with continuously rising memory or CPU usage. It covers how to confirm the problem, profile CPU and memory usage, investigate I/O and threads, and collect the information needed for further analysis or a Checkmk Support ticket.

 

Confirm the problem

Before profiling the process, confirm that the CPU or memory usage is actually increasing over time rather than experiencing a temporary spike.

Use ps to check the current resource usage of the affected process:

root@linux:$ ps -p <pid> -o pid,comm,%mem,rss,vsz,etime

 

To monitor the process continuously at five-second intervals, use:

root@linux:$ watch -n 5 "ps -p <pid> -o pid,comm,%mem,rss,vsz,etime"

 

You can also save the output to a file for later comparison:

root@linux:$ watch -n 5 "ps -p <pid> -o pid,comm,%mem,rss,vsz,etime | tee -a memory_log.txt"

 

The output contains the following relevant fields:

Field

Meaning

Field

Meaning

pid

Process ID

comm

Command name

%mem

Percentage of system RAM used by the process

rss

Resident Set Size, which is the physical memory currently used by the process in KB

vsz

Virtual memory size, which is the total reserved address space in KB

etime

Time elapsed since the process started


Look for a consistent trend rather than an individual high value:

  • RSS that climbs steadily without decreasing can indicate a memory leak or unbounded cache or queue growth.

  • CPU usage that remains close to 100% can indicate a busy loop, retry storm, or sustained workload.

  • The elapsed time can help correlate the behavior with an update, configuration change, or increase in workload.

 

Profile high CPU usage

For Python processes, use py-spy to inspect what the process is doing without stopping it.

py-spy dump takes a single snapshot of the process stack traces and is useful when investigating a process that appears to be stuck.

py-spy record profiles the process over a period of time and saves the result for further analysis.

 

Install the required tools

Use uv to install the Python-based troubleshooting tools used in this article.

Install uv:

root@linux:$ curl -LsSf https://astral.sh/uv/install.sh | sh

 

Then install smem2 and py-spy:

root@linux:$ uv tool install smem2 root@linux:$ uv tool install py-spy

uv tool install <package> installs a Python-based command-line tool permanently. Alternatively, uvx <package> runs a tool temporarily without installing it.

 

Monitor CPU usage with py-spy top

Use py-spy top to see which functions are currently consuming CPU:

root@linux:$ py-spy top --pid $(pgrep -f cmk-ui-job-scheduler)

 

To include subprocesses:

root@linux:$ py-spy top --pid $(pgrep -f cmk-ui-job-scheduler) --subprocesses

This provides an immediate overview of what the process is doing before recording a longer profile.

 

Take a stack snapshot with py-spy dump

Use py-spy dump to capture the call stack of every thread at a specific point in time:

root@linux:$ py-spy dump --pid $(pgrep -f cmk-ui-job-scheduler)

 

To also show local variables at each frame:

root@linux:$ py-spy dump --pid $(pgrep -f cmk-ui-job-scheduler) --locals

Run the command two or three times with a few seconds between each run.

 

If the same thread remains on the same line in each snapshot, this can indicate a hang or deadlock. If the location changes between snapshots, the process is likely still actively processing work.

 

Record CPU activity with py-spy

Use py-spy record to capture CPU activity over a longer period and generate a flamegraph:

root@linux:$ py-spy record -o profile.svg --pid $(pgrep -f cmk-ui-job-scheduler) --duration 300

 

To include subprocesses:

root@linux:$ py-spy record -o profile.svg --pid $(pgrep -f cmk-ui-job-scheduler) --duration 300 --subprocesses

 

To change the sampling rate:

root@linux:$ py-spy record -o profile.svg --pid $(pgrep -f cmk-ui-job-scheduler) --duration 60 --rate 50

Open the resulting profile.svg file in a web browser. Wider blocks in the flamegraph indicate that more time was spent in that function.

 

 

Create a speedscope profile (Optional)

Tip
Installing speedscope is optional if you intend to provide the generated file to Checkmk Support.

 

You can also create a profile in speedscope format:

root@linux:$ py-spy record -o profile.speedscope.json --pid $(pgrep -f cmk-ui-job-scheduler) --duration 300 --format speedscope

The resulting file can be provided directly to Checkmk Support for analysis.

 

If you want to analyze the file yourself, install speedscope:

root@linux:$ npm install -g speedscope

 

Then open the profile:

root@linux:$ speedscope profile.speedscope.json
image-20260914-133154.png

 

Investigate rising memory usage

If memory usage continues to increase, investigate it separately from CPU usage. Rising CPU usage can sometimes also be a symptom of increasing memory consumption, for example when garbage collection has to perform more work as the heap grows.

Check the process memory:

root@linux:$ cat /proc/$(pgrep -f cmk-ui-job-scheduler)/status | grep -E 'VmRSS|VmSize|VmSwap'

 

Check the number of open file descriptors:

root@linux:$ ls /proc/$(pgrep -f cmk-ui-job-scheduler)/fd | wc -l

 

Inspect the open files:

root@linux:$ lsof -p $(pgrep -f cmk-ui-job-scheduler)

A continuously increasing number of file descriptors can help identify a resource leak associated with the increasing memory usage.

For a more detailed memory overview, use smem:

root@linux:$ smem -P cmk-ui-job-scheduler

If the problem appears to be related specifically to the Python heap, a heap-focused profiler such as memray or application-generated tracemalloc snapshots can provide more useful information than py-spy, which primarily profiles CPU activity.

 

Check I/O activity

If neither CPU nor memory usage clearly explains the problem, check whether the process is waiting for disk or network I/O.

Summarize system calls and the time spent in them:

root@linux:$ strace -p $(pgrep -f cmk-ui-job-scheduler) -c

 

To inspect only network-related system calls:

root@linux:$ strace -p $(pgrep -f cmk-ui-job-scheduler) -f -e trace=network

 

To monitor disk I/O:

root@linux:$ iotop -p $(pgrep -f cmk-ui-job-scheduler)

This can help identify cases where the process appears to be under load but is actually spending significant time waiting for disk or network operations.

 

Check the process and its threads

Use the following command to display general information about the process, including its thread count and current state:

root@linux:$ ps -p $(pgrep -f cmk-ui-job-scheduler) -o pid,ppid,nlwp,stat,%cpu,%mem,etime,cmd

 

To list all threads:

root@linux:$ ps -eLf | grep $(pgrep -f cmk-ui-job-scheduler)

 

To monitor CPU usage for individual threads:

root@linux:$ top -H -p $(pgrep -f cmk-ui-job-scheduler)

 

Alternatively, use htop:

root@linux:$ htop -p $(pgrep -f cmk-ui-job-scheduler)

In htop, press H to toggle the display of individual threads.

 

Inspect the memory map

If RSS continues to increase, use pmap to determine which memory regions are growing.

Display the detailed memory map:

root@linux:$ pmap -x $(pgrep -f cmk-ui-job-scheduler)

 

To display only the totals:

root@linux:$ pmap -x $(pgrep -f cmk-ui-job-scheduler) | tail -1

 

You can compare two snapshots taken several minutes apart:

pmap -x $(pgrep -f cmk-ui-job-scheduler) > /tmp/pmap_before.txt sleep 300 pmap -x $(pgrep -f cmk-ui-job-scheduler) > /tmp/pmap_after.txt diff /tmp/pmap_before.txt /tmp/pmap_after.txt

Comparing the snapshots can help determine which memory region is growing, such as the heap, shared libraries, or mapped files.

 

Collect information for Checkmk Support

If you cannot identify the cause, collect the troubleshooting information before restarting the affected process or Checkmk site. Once the process has been restarted, the in-memory state used by tools such as py-spy, pmap, and lsof can no longer be collected.

Before opening a support ticket, collect:

  • One or more py-spy dump outputs.

  • The py-spy recording, such as profile.svg or profile.speedscope.json, if CPU usage is involved.

  • pmap -x output, ideally with snapshots taken several minutes apart if memory usage is increasing.

  • Multiple ps -o pid,comm,%mem,rss,vsz,etime readings showing the trend over time.

  • lsof -p <pid> output if file descriptor growth was observed.

  • The Checkmk version, site name, approximate time when the behavior began, and any relevant recent update, configuration change, or increase in workload.

 

Package the collected files together and include them with the support ticket. This gives Checkmk Support information about how the process behaved over time rather than only a single snapshot.

 

Example troubleshooting workflow

The following example shows a complete troubleshooting sequence for cmk-ui-job-scheduler:

ps -p $(pgrep -f cmk-ui-job-scheduler) -o pid,comm,%mem,rss,vsz,etime py-spy top --pid $(pgrep -f cmk-ui-job-scheduler) py-spy dump --pid $(pgrep -f cmk-ui-job-scheduler) py-spy record -o profile.svg --pid $(pgrep -f cmk-ui-job-scheduler) --duration 300 py-spy record -o profile.speedscope.json --pid $(pgrep -f cmk-ui-job-scheduler) --duration 300 --format speedscope py-spy top --pid $(pgrep -f cmk-ui-job-scheduler) py-spy dump --pid $(pgrep -f cmk-ui-job-scheduler) ps -p $(pgrep -f cmk-ui-job-scheduler) -o pid,comm,%mem,rss,vsz,etime smem -P cmk-ui-job-scheduler

Start with ps to establish a baseline. Then use py-spy top and py-spy dump to characterize the process before recording a five-minute profile. After the recording, run py-spy top and py-spy dump again and take a final ps reading to compare the CPU and memory usage with the initial baseline.

 

 

 

Related articles