Troubleshooting “source code string cannot contain null bytes” during activate changes

Troubleshooting “source code string cannot contain null bytes” during activate changes

This failure can occur if there was a storage-related issue (e.g., unavailability of the storage backend, or a read-only file system).  

LAST TESTED ON CHECKMK 2.3.0P1

Table of Contents

Problem

The activation of changes doesn't work properly, yielding this error message:

Error

source code string cannot contain null bytes


And the crash report containing a traceback like this:

  File "/omd/sites/mysite/lib/python3/cmk/gui/pages.py", line 101, in handle_page
    action_response = self.page()
  File "/omd/sites/mysite/lib/python3/cmk/gui/wato/pages/activate_changes.py", line 974, in page
    activation_id = manager.start(
  File "/omd/sites/mysite/lib/python3/cmk/gui/watolib/activate_changes.py", line 767, in start
    execute_activation_cleanup_background_job(maximum_age=60)
  File "/omd/sites/mysite/lib/python3/cmk/gui/watolib/activate_changes.py", line 1417, in execute_activation_cleanup_background_job
    if job.is_active():
  File "/omd/sites/mysite/lib/python3/cmk/gui/background_job.py", line 529, in is_active
    job_status = self.get_status()
  File "/omd/sites/mysite/lib/python3/cmk/gui/background_job.py", line 622, in get_status
    status = self._jobstatus_store.read()
  File "/omd/sites/mysite/lib/python3/cmk/gui/background_job.py", line 819, in read
    if not (raw_status_spec := self.read_raw()):
  File "/omd/sites/mysite/lib/python3/cmk/gui/background_job.py", line 844, in read_raw
    status: dict[str, Any] = store.load_object_from_file(
  File "/omd/sites/mysite/lib/python3/cmk/utils/store/__init__.py", line 186, in load_object_from_file
    return ObjectStore(Path(path), serializer=DimSerializer()).read_obj(default=default)
  File "/omd/sites/mysite/lib/python3/cmk/utils/store/_file.py", line 136, in read_obj
    return self._serializer.deserialize(raw) if raw else default
  File "/omd/sites/mysite/lib/python3/cmk/utils/store/_file.py", line 75, in deserialize
    return literal_eval(raw.decode("utf-8"))
  File "/omd/sites/mysite/lib/python3.11/ast.py", line 64, in literal_eval
    node_or_string = parse(node_or_string.lstrip(" \t"), mode='eval')
  File "/omd/sites/mysite/lib/python3.11/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,

Solution


  1. Log into your central Checkmk site as the site user

    OMD[mysite] omd stop


    .

  2. Navigate to the Background jobs directory:

    OMD[mysite]:cd $OMD_ROOT/var/check_mk/background_jobs


    .

  3. List out the files in this directory and then delete them

    OMD[mysite]:$OMD_ROOT/var/check_mk/background_jobs$ ls -l


    .

  4. Start the site again:

    OMD[mysite] omd start