This article explains how to troubleshoot DCD issues, reproduce REST API errors manually, and collect the information needed for further investigation.

LAST TESTED ON CHECKMK 2.4.0P1

Table of Contents

1 General debug options
2 Problem
3 Solution
- 3.1 Test the REST API manually
- 3.2 Check the logs
- 3.3 Check custom attributes
4 Related articles

General debug options

Increase the log level

Follow the instructions in How to collect troubleshooting data for various issue types to increase the log level of the DCD.

Manual execution

Since Checkmk 2.1, the DCD uses the REST API. When troubleshooting a DCD issue, you can use the REST API request shown in the logs to determine whether the problem also occurs independently of the DCD.

For example:

2022-05-13 13:37:35,970 [10] [cmk.dcd.web_api] > [GET] http://localhost:5005/mysite/check_mk/api/1.0/domain-types/host_config/collections/all

Problem

Internal server error

When running the DCD, you may encounter an Internal Server Error.

The error can look similar to the following:

ERROR An exception occurred
Traceback (most recent call last):
  File "/omd/sites/mysite/lib/python3/cmk/cee/dcd/connectors.py", line 229, in _execute_phase2
    cmk_hosts = self._web_api.get_all_hosts()
  File "/omd/sites/mysite/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
    ...
requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url:
http://localhost:5005/mysite/check_mk/api/1.0/domain-types/host_config/collections/all

Opening the affected REST API URL directly in the browser can provide additional information about the error. In this example, the response identifies the custom fields lat and long as unknown fields.

In other cases, dcd.log might contain an error similar to the following:

2026-08-03 08:26:53,565 [10] [cmk.dcd.ConnectionManager] Expected cycle start: 1785738443.565528
2026-08-03 08:26:53,696 [10] [cmk.dcd.Manager] 1 batches arrived 50, last activated is reset
2026-08-03 08:26:53,769 [40] [cmk.dcd.Manager] Error during processing:
('500 Server Error: INTERNAL SERVER ERROR for url:
http://localhost:5001/SITE/check_mk/api/internal/domain-types/host_config/collections/all',)

AttributeError: 'NoneType' object has no attribute 'get'.
Crash report generated. Please submit.
For further details please check web.log

The error indicates that the DCD encountered a problem while retrieving host configuration data through the REST API.

Solution

Test the REST API manually

Test the REST API independently of the DCD to determine whether retrieving the host configuration produces the same error.

Run the following command on the Checkmk server:

curl -sk -X GET \
"https://localhost/<SITE_NAME>/check_mk/api/1.0/domain-types/host_config/collections/all?effective_attributes=true" \
-H "Authorization: Bearer <username> <secret>" \
-H "Accept: application/json" \
-o /tmp/all_hosts_dump.json \
-w "\nHTTP %{http_code}\n"

Replace <SITE_NAME>, <username>, and <secret> with the appropriate values for your environment.

The command retrieves the host configuration, including the effective attributes, and saves the response to:

/tmp/all_hosts_dump.json

The HTTP status code is printed after the request completes.

If the request returns an HTTP 500 error, check web.log for additional information.

Check the logs

Check the following logs for errors occurring at the same time as the failed REST API request:

~/var/log/dcd.log
~/var/log/web.log

Note
If dcd.log reports Crash report generated. Please submit., include the generated crash report when opening a Checkmk support ticket. The crash report contains additional diagnostic information that can help Support investigate the underlying error.

Check custom attributes

Custom host attributes can cause problems when the DCD retrieves host configuration data.

In one known case, the wato_geo_fields.py WATO plug-in from the Checkmk Treasures caused the issue.

As a workaround, remove the custom values and run the DCD again.