Troubleshooting "invalid checker timestamp" (1970) error
Problem
For some reason, it could be the case that you will see invalid checker timestamps in the cmc.log and the Checkmk GUI:
2022-04-28 11:45:33 [4] [core 25015] [smartping scheduler] refuse to schedule invalid jobhost "testhost-a" at 1970-01-01 01:00:00 with an invalid timestamp 2022-04-28 11:45:33 [4] [core 25015] [smartping scheduler] refuse to schedule invalid jobhost "testhost-b" at 1970-01-01 01:00:00 with an invalid timestamp 2022-04-28 11:45:33 [4] [core 25015] [smartping scheduler] refuse to schedule invalid jobhost "testhost-c" at 1970-01-01 01:00:00 with an invalid timestamp
In <2.1 there is no way to debug that due to the format of the state file. With Checkmk 2.1, the state file will have a different format. Anyway: If you face that issue in <2.1, you can open a case for that.
If you face this issue <=2.1, you can follow our Workaround or open a support ticket and provide the following information by using our support diagnostic.
Workaround
Log on to the affected site.
root@mylinuxhost~# su - mysite
.
navigate to ~/var/check_mk/core
OMD[mysite]:~$ cd ~/var/check_mk/core/
.
here you will find all the "state" files
OMD[mysite]:~/var/check_mk/core$ ls archive/ helper_config/ state.1 state.5 state.9 autochecks/ history state.2 state.6 config state state.3 state.7 discovered_host_labels/ state-version-7 state.4 state.8
The cmc is always using the file "state"
.Now rename the "state" file to state.(number)
BE AWARE: all ad hoc Downtimes + Acknowledgements + Comments are gone, but you can restore the old state file later.
The state is created every 10 minutes or after changes in the GUI.
A backup is created when the new file is 5% smaller than the old one. Checkmk will save 30 state files and then overwrite the old ones.
OMD[mysite]:~/var/check_mk/core$ mv state state.10
The number depends on how many state.(number) files you already have.
.
You can either restart the whole site with
OMD[mysite]:~ omd restart
or do
OMD[mysite]:~ cmk -O
Now a new "state" file is created!
Restore a state file
mv state.NN state
Don't restart the core in that case!
Related articles