Debug DCD issues

If you're running into a DCD issue, you can follow these steps to get a better overview of that issue.

LAST TESTED ON CHECKMK 2.3.0P1

Table of Contents

General debug options

Increase the log level

Follow that manual to increase the log level of the DCD: How to collect troubleshooting data for various issue types#DCD

Manual execution

As described in Debug DCD issues#Solution, you can grab the URLs and try them inside a script or via the browser, e.g.:

Since check 2.1, the DCD uses the REST API. Checkmk 1.6 is using the WEB-API to create a host (Checkmk REST API)

2022-05-13 13:37:35,970 [10] [cmk.dcd.web_api] > [GET] http://localhost:5005/mysite/check_mk/api/1.0/domain-types/host_config/collections/all

2022-05-13 13:38:28,917 [10] [cmk.dcd.web_api] > [POST] http://localhost:5005/mysite/check_mk/api/1.0/domain-types/host_config/actions/bulk-create/invoke ((), {'data': None, 'json': {'ent
ries': [{'host_name': 'aws', 'folder': '/', 'attributes': {'locked_by': {'site_id': 'mysite', 'program_id': 'dcd', 'instance_id': 'connection_1'}, 'locked_attributes': ['cb', 'labels', 's
ite', 'tag_address_family', 'tag_agent'], 'site': 'mysite', 'labels': {'ds': 'pgb'}, 'tag_address_family': 'no-ip', 'tag_agent': 'no-agent', 'cb': 'hh'}}]}, 'params': {'bake_agent': 'fals
e'}})

2022-05-13 13:38:28,917 [10] [cmk.dcd.web_api] > [POST] http://localhost:5005/mysite/check_mk/api/1.0/domain-types/host_config/actions/bulk-create/invoke ((), {'data': None, 'json': {'ent
ries': [{'host_name': 'aws', 'folder': '/', 'attributes': {'locked_by': {'site_id': 'mysite', 'program_id': 'dcd', 'instance_id': 'connection_1'}, 'locked_attributes': ['cb', 'labels', 's
ite', 'tag_address_family', 'tag_agent'], 'site': 'mysite', 'labels': {'ds': 'pgb'}, 'tag_address_family': 'no-ip', 'tag_agent': 'no-agent', 'cb': 'hh'}}]}, 'params': {'bake_agent': 'fals
e'}})

Problem

Internal Server Error

When running the DCD, you may face the following issue:

Screenshot of Dynamic host management. Error during phase 2.2. Fetching existing hosts

ERROR An exception occurred
Traceback (most recent call last):
	File “/omd/sites/mysite/lib/python3/cmk/cee/dcd/connectors.py:, line 229, in _execute_phase2
		Cmk hosts = self._web_api.get_all_hosts()
	File “/omd/sites/mysite/lib/python3.9/site-packages/request/models.py:, line 960, in raise_for_status…
		Raise HTTPError (http_error_msg, response=self)
request.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url:
http://localhost:5005/mysite/check_mk/api/1.0/domain-types/host_config/collections/all

Solution

Grab the URL and paste it into the browser.

Screenshot of browser window of error Server was about to send an invalid response.

In that case, custom attributes were used, which caused trouble. In this specific case, the wato plugin "wato_geo_fields.py" from the Checkmk treasures was used.

A workaround would be to remove all custom values and run the DCD again. This issue is already reported internally.