/
Troubleshooting "ERROR: Metric backend did not become ready within 30 seconds, aborting"

Troubleshooting "ERROR: Metric backend did not become ready within 30 seconds, aborting"

When restoring under a new name, copying or moving a site, the command fails during post-restore script execution.

LAST TESTED ON CHECKMK 2.5.0P12

Table of Contents

Problem

When restoring a site under a new name while the original source site remains running, the omd restore mynewsite /path/to/mysite.tar.gz command fails during post-restore script execution.
The same error can also occur upon copying or renaming a site (omd cp mysite mynewsite  or omd mv mysite mynewsite).
This only affects Checkmk Ultimate and Checkmk UltimateMT, version 2.5 and above, with the metric backend enabled.

The command output displays the following error:

Executing post-restore script "03_recover-metric-backend-schema"... -| INFO: No config file provided, using default config values. -| INFO: Starting metric backend for schema upgrade... -| ERROR: Metric backend did not become ready within 30 seconds, aborting. Check /omd/sites/mynewsite/var/log/clickhouse-server/clickhouse-server-startup.log for details. ERROR (exit code: 1) Error in non-priviledged sub-process.

 

Inspecting /omd/sites/mysite/var/log/clickhouse-server/clickhouse-server-startup.log shows a port collision exception:

Code: 210. DB::Exception: Listen [127.0.0.1]:25382 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 127.0.0.1:25382

 

Cause

While omd restore/cp/mv  reassigns conflicting network ports for standard services (such as Apache, Livestatus, and RabbitMQ) when renaming a site, they do not automatically reassign the metric backend port.
Because the original source site is still running and bound to the same port, the newly restored site fails to initialize its metric backend schema upgrade. 


Solution

To complete the site restore, copy or move while avoiding port conflicts:

  1. Temporarily stop the running source site: sudo omd stop mysite

  2. Rerun the command you ran before.

  3. Change the metric backend port configuration for the new site, before starting the source site again.

Note: If a previous failed attempt left incomplete site files or user permissions behind, remove the broken target site before re-running the restore: sudo omd rm mynewsite 


Related articles