Troubleshooting Apache service not starting

Owned by Anastasios Thomaidis
Last updated: Dec 11, 2023 by Matthew Hierholzer

This article details how to troubleshoot apace not starting on a Checkmk server.

LAST TESTED ON CHECKMK 2.0.0P1

Table of Contents


We do not support any modification of our config files (e.g., apache, .profile, etc.). If you do that and the site is broken, you will run out of support, especially on the Checkmk Appliance.

We saw the bottom error several times, especially when dealing with check_sql.

Error Messages

Starting apache...apache2: Syntax error on line 236 of /omd/sites/mysite/etc/apache/apache.conf
libpython3.9.so.1.0: cannot open shared object file: No such file or directory

Problem

The default for LD_LIBARY_PATH should be:

LD_LIBRARY_PATH=$OMD_ROOT/local/lib:$OMD_ROOT/lib

But for some reason, the .profile file of the site user seems to be broken, and after the restart of the site, you will face the following issue:

Starting apache...apache2: Syntax error on line 236 of /omd/sites/mysite/etc/apache/apache.conf: Syntax error on line 2 of /omd/sites/mysite/etc/apache/conf.d/01_wsgi.conf: Cannot load /omd/sites/mysite/lib/apache/modules/mod_wsgi.so into server: libpython3.9.so.1.0: cannot open shared object file: No such file or directory
..........ERROR

Solution

  1. Check if the env is right

    su - mysite  
OMD[mysite]:~$ echo $LD_LIBRARY_PATH
/omd/sites/mysite/opt/oracle/instantclient_21_4

    In that case, the env is not correct. The env is pointing only to the libraries of the Oracle instantclient

  2. Check if the library is available.

    OMD[mysite]:~$ find -L -name "*libpython3.9.so.1.0*"
./version/lib/libpython3.9.so.1.0
./lib/libpython3.9.so.1.0

    In that case, the library is only available in the origin and default path of LD_LIBRARY_PATH

  3. Restore the default variable

    You will find the original files of .profile or .bashrc in "/omd/versions/2.1.0b7.cee/skel"

    ➜ /omd/versions/2.1.0b7.cee/skel ls -altr
insgesamt 36
-rw-r--r-- 1 root root 2055 Apr 25 00:02 .profile
-rw-r--r-- 1 root root 1091 Apr 25 00:02 .bashrc

    One way would be to copy the LD_LIBRARY_PATH value from .profile and put it directly to the .profile of the site user:

    su - mysite
OMD[mysite]:~$ vi .profile


    To make the changes permanent, please enter:

    OMD[mysite]:~$ source .profile

  4. set the LD_LIBARY_PATH the right way

    extend the LD_LIBRARY_PATH like that ":path/to/oracle/instanclient"

    OMD[mysite]:~$ vi .profile 
OMD[mysite]:~$ source .profile


    In the end, it should look like this:

    OMD[mysite]:~$ echo $LD_LIBRARY_PATH
/omd/sites/mysite/local/lib:/omd/sites/mysite/lib:path/to/oracle/instanclient
  5. Now after a restart of the site Apache the error should be gone!