How-to debug failed backup jobs

Problem

Basic information about mkbackup

After configuring the backup job, a cronjob is created. This job can be inspected on the command line, after logging in via SSH as site user:

If you need more debugging, you can add --verbose and --debug to the mkbackup command:

Collection of error messages

Error 1

Failed to perform a backup: [Errno 104] Connection reset by peer

Solution

Find the correct backup job

Please run the backup directly on the command line and forward the output to a log file.

Here it looks like Checkmk is using pnp4nagios instead of Round Robin Database (RRD). We recommend converting the performance data to the RRD format. Please follow the steps described here: Customizing the RRD structure

Don't forget to stop the site before converting the files!

Now the backup should run without any errors.

Error 2

UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 122: surrogates not allowed

Solution

Please run the backup directly on the command line and forward the output to a log file.

Let's check the log now:

This issue is that this file contains a non-ascii character at the end. "AUTORIT�.rrd"

To correct this, we must delete or rename this file. The safest solution would be to rename it.

Advanced Debugging

Tracing backup with strace

If a backup fails or behaves unexpectedly and the normal backup output does not provide enough detail, you can use strace to trace which files and directories are accessed during the backup. This is useful for identifying permission issues, unexpected paths, or filesystem-related problems.

Run the backup with strace enabled

Switch to the Checkmk site user and change to the system /tmp directory.
Do not use the site’s ~/tmp directory.

Run the backup command wrapped with strace:

Replace yourmybackupjobname with the name of the backup job you want to analyze.

This command will:

• Trace all system calls made during the backup, including child processes

• Write the full trace output to /tmp/strace_backup.log

• Run the backup in verbose mode

• Disable compression to simplify analysis

• Store the standard backup output in backup.log

Using -s 4096 with strace

The -s option in strace defines the maximum string size that will be printed in the output.

By default, strace only shows the first 32 characters of a string. This often results in truncated file paths, SQL queries, or command arguments, which can make troubleshooting more difficult.

Using -s 4096 increases the maximum displayed string length to 4096 characters. This is especially helpful when debugging backup issues because you can see:

• Full file paths being accessed

• Complete SQL queries

• Full command line arguments

• Complete data being sent or received

Example with full string output (recommended for detailed debugging)

This command provides much more detailed output, which is useful when investigating complex or unclear backup problems.

Example without -s 4096 (smaller output file)

This will still work, but strings will be truncated to 32 characters.

Important note
Using -s 4096 can produce a very large output file, especially on busy or large sites. If disk space is limited or the backup runs for a long time, customers may prefer to:

• Run without -s 4096 first

• Only rerun with -s 4096 if more detail is required

In many cases, the default output is sufficient. Use -s 4096 when you specifically need full string visibility for deeper analysis.

List directories accessed during the backup

To extract a list of directories accessed during the backup, run:

This will produce a unique, sorted list of Checkmk site directories that were opened during the backup process.

When this is useful

This method can be helpful in situations such as:

• Backups failing without clear error messages

• Backups including unexpected data

• Backups missing expected files or directories

• Suspected filesystem or permission issues

Cleanup

After completing your analysis, remove the trace file to free disk space:

Related articles