Random Reboots of my Ubuntu Server NAS, Gaming-Server no matter what

[Neocult]

Host: ASRock B550 Phantom Gaming-ITX/ax 2023 - desktop with Ubuntu 22.04 (exported from Linux-Hardware.org)

PROBE ID

Hi everyone,

I am about to become desperate with my NAS / Gaming Server machine, which I have now reworked for over a year by now. I have replaced by now almost every component in it and I just can´t make it stable. I have random reboots without kernel panics, crash logs ... and by now I have even a UPS installed, just in case ...

I have swapped the power supply, the mainboard to a newer one. Bios Updates on previous and current Board, haven´t helped. I installed Ubuntu 22.04.1 LTS ... tried updating to 22.10 ... tried bleeding edge kernel ... and a complete do-over with a fresh install seem to be the same issue. I get all green on boot. I tried to get newer linux-firmware to run, by now I even lifted the mainboard out of the case, disconnected HDDs and other attachments just in case (even though no errors at all ... didn´t help.

I ran memtest+ for hours on end, it didn´t report anything. I tried stress test and stress-ng with the verify option and it tells me the cpu is fine... ECC is running ... but If I deactivate it in BIOS the same issues still appear.

If someone can point to something, I take all recommendations and will give them a test. Maybe I overlooked something very simple in the logs.

Looking forward to any hint. Thanks for reading!

Devices on the board are the following:

DEVICE	STATUS	COMMENT
BUS: PCI ID: 1002:1636:1002:1636 CLASS: 03-00 VENDOR: Advanced Micro Devices, Inc. [AMD/ATI] DEVICE: Renoir TYPE: graphics card DRIVER: amdgpu	Works
BUS: PCI ID: 1002:1637:1002:1637 CLASS: 04-03 VENDOR: Advanced Micro Devices, Inc. [AMD/ATI] DEVICE: Renoir Radeon High Definition Audio Controller TYPE: sound DRIVER: snd_hda_intel	Detected
BUS: PCI ID: 1022:15e3:1849:2228 CLASS: 04-03 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Family 17h/19h HD Audio Controller TYPE: sound DRIVER: snd_hda_intel	Detected
BUS: PCI ID: 8086:15f3:8086:0000 CLASS: 02-00 VENDOR: Intel Corporation DEVICE: Ethernet Controller I225-V TYPE: network DRIVER: igc	Works
BUS: PCI ID: 8086:2723:8086:0084 CLASS: 02-80 VENDOR: Intel Corporation DEVICE: Wi-Fi 6 AX200 TYPE: network DRIVER: iwlwifi	Detected
BUS: PCI ID: 1022:43eb:1b21:1062 CLASS: 01-06-01 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: 500 Series Chipset SATA Controller TYPE: storage DRIVER: ahci	Works
BUS: PCI ID: 1cc1:8201:1cc1:8201 (2x) CLASS: 01-08-02 VENDOR: ADATA Technology Co., Ltd. DEVICE: XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive TYPE: storage DRIVER: nvme	Works
BUS: PCI ID: 1022:1448 CLASS: 06-00 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Renoir Device 24: Function 0 TYPE: bridge DRIVER: -	Detected
BUS: PCI ID: 1022:1449 CLASS: 06-00 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Renoir Device 24: Function 1 TYPE: bridge DRIVER: -	Detected
BUS: PCI ID: 1022:144a CLASS: 06-00 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Renoir Device 24: Function 2 TYPE: bridge DRIVER: -	Detected
BUS: PCI ID: 1022:144b CLASS: 06-00 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Renoir Device 24: Function 3 TYPE: bridge DRIVER: k10temp	Detected
BUS: PCI ID: 1022:144c CLASS: 06-00 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Renoir Device 24: Function 4 TYPE: bridge DRIVER: -	Detected
BUS: PCI ID: 1022:144d CLASS: 06-00 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Renoir Device 24: Function 5 TYPE: bridge DRIVER: -	Detected
BUS: PCI ID: 1022:144e CLASS: 06-00 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Renoir Device 24: Function 6 TYPE: bridge DRIVER: -	Detected
BUS: PCI ID: 1022:144f CLASS: 06-00 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Renoir Device 24: Function 7 TYPE: bridge DRIVER: -	Detected
BUS: PCI ID: 1022:1630:1022:1630 CLASS: 06-00 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Renoir/Cezanne Root Complex TYPE: bridge DRIVER: -	Detected
BUS: PCI ID: 1022:1632 (3x) CLASS: 06-00 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Renoir PCIe Dummy Host Bridge TYPE: bridge DRIVER: -	Detected
BUS: PCI ID: 1022:1634:1022:1453 (2x) CLASS: 06-04 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Renoir PCIe GPP Bridge TYPE: bridge DRIVER: pcieport	Works
BUS: PCI ID: 1022:1635:1022:1635 CLASS: 06-04 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Renoir Internal PCIe GPP Bridge to Bus TYPE: bridge DRIVER: pcieport	Works
BUS: PCI ID: 1022:43e9:1b21:0201 CLASS: 06-04 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: PCI bridge TYPE: bridge DRIVER: pcieport	Works
BUS: PCI ID: 1022:43ea:1b21:3308 (3x) CLASS: 06-04 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: PCI bridge TYPE: bridge DRIVER: pcieport	Works
BUS: PCI ID: 1022:790e:1849:ffff CLASS: 06-01 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: FCH LPC Bridge TYPE: bridge DRIVER: -	Detected
BUS: PCI ID: 1022:15df:1022:15df CLASS: 10-80 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Family 17h (Models 10h-1fh) Platform Security Processor TYPE: encryption controller DRIVER: ccp	Detected
BUS: PCI ID: 1022:790b:1849:ffff CLASS: 0c-05 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: FCH SMBus Controller TYPE: smbus DRIVER: i2c_piix4	Detected
BUS: PCI ID: 1022:1639:1849:ffff (2x) CLASS: 0c-03-30 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: Renoir/Cezanne USB 3.1 TYPE: usb controller DRIVER: xhci_pci	Detected
BUS: PCI ID: 1022:43ee:1b21:1142 CLASS: 0c-03-30 VENDOR: Advanced Micro Devices, Inc. [AMD] DEVICE: 500 Series Chipset USB 3.1 XHCI Controller TYPE: usb controller DRIVER: xhci_pci	Detected
BUS: USB ID: 8087:0029 CLASS: e0-01-01 VENDOR: Intel Corp. DEVICE: AX200 Bluetooth TYPE: bluetooth DRIVER: btusb	Detected
BUS: USB ID: 174c:2074 CLASS: 09-00-01 VENDOR: ASMedia Technology Inc. DEVICE: ASM1074 High-Speed hub TYPE: hub DRIVER: hub	Detected
BUS: USB ID: 174c:3074 CLASS: 09-00-00 VENDOR: ASMedia Technology Inc. DEVICE: ASM1074 SuperSpeed hub TYPE: hub DRIVER: hub	Detected
BUS: USB ID: 1d6b:0002 (3x) CLASS: 09-00-00 VENDOR: Linux Foundation DEVICE: 2.0 root hub TYPE: hub DRIVER: hub	Detected
BUS: USB ID: 1d6b:0003 (3x) CLASS: 09-00-00 VENDOR: Linux Foundation DEVICE: 3.0 root hub TYPE: hub DRIVER: hub	Detected
BUS: USB ID: 26ce:01a2 CLASS: 03-00-00 VENDOR: ASRock DEVICE: LED Controller TYPE: human interface DRIVER: usbhid	Detected
BUS: USB ID: 062a:4101 CLASS: 03-01-01 VENDOR: MosArt Semiconductor Corp. DEVICE: Wireless Keyboard/Mouse TYPE: keyboard DRIVER: usbhid	Detected
BUS: USB ID: 0463:ffff CLASS: 03-00-00 VENDOR: MGE UPS Systems DEVICE: UPS TYPE: ups DRIVER: usbhid	Detected
BUS: EISA ID: aoc-aoc2702 VENDOR: AOC DEVICE: Q27G2WG4 AOC2702 2560x1440 597x336mm 27.0-inch TYPE: monitor DRIVER: -	Detected
BUS: SYS ID: american-megatrends-l2-61-02-14-2023 VENDOR: American Megatrends International, LLC. DEVICE: BIOS L2.61 02/14/2023 TYPE: bios DRIVER: -	Works
BUS: SYS ID: amd-23-96-1-ryzen-7-pro-4750g-with-radeon-graphics (16x) VENDOR: AMD DEVICE: Ryzen 7 PRO 4750G with Radeon Graphics TYPE: cpu DRIVER: -	Works
BUS: SYS ID: kingston-9965745-032-a00g-dimm [13C] VENDOR: Kingston DEVICE: RAM 9965745-032.A00G 16GB DIMM DDR4 2400MT/s TYPE: memory DRIVER: -	Works
BUS: SYS ID: kingston-9965745-032-a00g-dimm [57E] VENDOR: Kingston DEVICE: RAM 9965745-032.A00G 16GB DIMM DDR4 2400MT/s TYPE: memory DRIVER: -	Works
BUS: SYS ID: asrock-b550-phantom-gaming-itx-ax VENDOR: ASRock DEVICE: Motherboard B550 Phantom Gaming-ITX/ax TYPE: motherboard DRIVER: -	Works
BUS: IDE ID: samsung-ssd-870-qvo-2tb [E9F] VENDOR: Samsung DEVICE: SSD 870 QVO 2TB TYPE: disk DRIVER: ahci, sd	Works
BUS: NVME ID: adata-sx8200pnp-2tb [3C8] VENDOR: ADATA DEVICE: SX8200PNP 2TB TYPE: disk DRIVER: nvme	Works
BUS: NVME ID: adata-sx8200pnp-2tb [921] VENDOR: ADATA DEVICE: SX8200PNP 2TB TYPE: disk DRIVER: nvme	Works

[Neocult]

So I found the issue after I tested more or less randomly for everything that I Could find. I swapped all components and the CPU is at fault. (A new 5600g operated without issues). My 4750G seems to have a faulty C6 state that is not fixed by newest BIOS / AGESA / Microcode update, neither on Gigabyte B550I Aorus Pro AX nor ASRock B550i Phantom Gaming AC.

It ran stable on Windows because C6 is not enabled by default. If anyone else have the C6 issue I can recommend zenstates (https://github.com/r4m0n/ZenStates-Linux  with the --disable-c6 option). Another one is adding a kernel parameter to your boot options (processor.max_cstate=5) .... you can put that to /etc/default/grub into the CMD_LINE and/or CMD_LINE_DEFAULT (just added with a space to the other options) and then run "update-grub".

Because it is a CPU Crash in a specific deep sleep energy saving state the errors didn´t occur in Windows (not supporting it by default), a stress test of all cores would never reveal it (no sleeping cores), and if you run a memtest, uefi-screen or a performance-oriented application like proxmox it never switches to the faulty state.... as the CPU shuts down from a faulty state is immediately crashes and never left a trail in the logs ...