Troubleshooting Kernel Panic on rack4 Mark III

Owned by Matthew Hierholzer
Apr 25, 2024

On the Checkmk Appliance rack4 Mark III, upon reboot is it possible, that one sees a Kernel Panic.

AFFECTS CHECKMK HARDWARE APPLIANCE RACK4 MARK III WITH VERSION 1.7.0 AND ABOVE


Table of Contents

Problem

On the Checkmk Appliance rack4 Mark III, upon reboot is it possible, that one sees a Kernel Panic similar to the following.

[528.576055] {1} [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 5
[528.576062] {1} [Hardware Error]: event severity: fatal
[528.576068] {1} [Hardware Error]:  Error 0, type: fatal
[528.576074] {1} [Hardware Error]:   section_type: PCIe error
[528.576076] {1} [Hardware Error]:   port_type: 0, PCIe end point
[528.576081] {1} [Hardware Error]:   version: 3.0
[528.576084] {1} [Hardware Error]:   command: 0x0002, status: 0x0010
[528.576089] {1} [Hardware Error]:   device_id: 0000:01:00.1
[528.576093] {1} [Hardware Error]:   slot : 0
[528.576094] xhci_hcd 0000:00:14.0: xHCI host controller not responding, assume dead
[528.576096] {1} [Hardware Error]:   secondary_bus: 0x00
[528.576100] {1} [Hardware Error]:   vendor_id: 0x14e4, device_id: 0x165f
[528.576105] {1} [Hardware Error]:   class_code: 020000
[528.576109] {1} [Hardware Error]:   aer_uncor_status: 0x00100000, aer_uncor_mask: 0x00010000
[528.576115] {1} [Hardware Error]:   aer_uncor_severity: 0x000ef030
[528.576118) {1} [Hardware Error]:   TLP Header: 40000001 000002ef 90028090 00000000
[528.576127] Kernel panic - not syncing: Fatal hardware error!
[528.576132] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.1.0-18-amd64 #1 Debian 6.1.76-1
[528.576144] Hardware name: /01YM03, BIOS 2.20.1 09/13/2023
[528.576146] Call Trace:

[...]

Note, that this issue does not occur on shutdown.

Solution

There is a fix already committed to the Linux Kernel, but it has yet to make its way to the Debian version, on which the Checkmk Appliance is based.
The issue has no impact whatsoever on the appliance, so it can safely be ignored for now. Just make sure, it is the exact same error, that you are seeing.