Troubleshooting "invalid checker timestamp" (1970) error

Owned by Anastasios Thomaidis
Last updated: Dec 11, 2023 by Matthew Hierholzer
2 min read

This article explains how to troubleshoot an error of "invalid timestamp" within Checkmk.

LAST TESTED ON CHECKMK 2.0.0P1

Table of Contents

Problem

For some reason, it could be the case that you will see invalid checker timestamps in the cmc.log and the Checkmk gui:

2022-04-28 11:45:33 [4] [core 25015] [smartping scheduler] refuse to schedule invalid jobhost "testhost-a" at 1970-01-01 01:00:00 with an invalid timestamp
2022-04-28 11:45:33 [4] [core 25015] [smartping scheduler] refuse to schedule invalid jobhost "testhost-b" at 1970-01-01 01:00:00 with an invalid timestamp
2022-04-28 11:45:33 [4] [core 25015] [smartping scheduler] refuse to schedule invalid jobhost "testhost-c" at 1970-01-01 01:00:00 with an invalid timestamp

In <2.1 there is no way to debug that due to the format of the state file. With Checkmk 2.1, the state file will have a different format. Anyway: If you face that issue in <2.1, you can open a case for that.


If you face this issue <=2.1, you can follow our Workaround or open a support ticket and provide the following information by using our support diagnostic.

Screenshot of the Support Diagnostics page. CMC is enabled with Core files selected.


Workaround

  1. Log on to the affected site.

    root@mylinuxhost~# su - mysite

    .

  2. navigate to ~/var/check_mk/core

    OMD[mysite]:~$ cd ~/var/check_mk/core/

    .

  3. here you will find all the "state" files

    OMD[mysite]:~/var/check_mk/core$ ls
archive/                 helper_config/   state.1  state.5  state.9
autochecks/              history          state.2  state.6
config                   state            state.3  state.7
discovered_host_labels/  state-version-7  state.4  state.8

    The cmc is always using the file "state"
    .

  4. Now rename the "state" file to state.(number)

    BE  AWARE: all ad hoc Downtimes + Acknowledgements + Comments are gone, but you can restore the old state file later.

    The state is created every 10 minutes or after changes in the GUI.

    A backup is created when the new file is 5% smaller than the old one. Checkmk will save 30 state files and then overwrite the old ones.

    OMD[mysite]:~/var/check_mk/core$ mv state state.10

    The number depends on how many state.(number) files you already have.

    .

  5. You can either restart the whole site with

    OMD[mysite]:~ omd restart

    or do

    OMD[mysite]:~ cmk -O

    Now a new "state" file is created!

Restore a state file

Go back to Step 4 of the Troubleshooting "invalid checker timestamp" (1970) error#Solution and move the state.NN to state
mv state.NN state

Don't restart the core in that case!