User Tools

Site Tools


AMD Epyc Microcode Update on Boot

Problem:: Server reboots (sometimes crashes) unexpected. See Discussion at AMD Community :
“…AMD has identified an issue with the Linux cpuidle subsystem whereby a system using a newer kernel(4.13 or newer) with SMT enabled (BIOS default) and global C state control enabled (also BIOS default) may exhibit an unexpected reboot. The likelihood of this reboot is correlated with the frequency of idle events in the system. AMD has released updated system firmware to address this issue. Please contact your system provider for a status on this updated system firmware. Prior to the availability of this updated system firmware, you can work around the issue with the following option:
Boot the kernel with the added command line option idle=nomwait …”

# edit /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="nomodeset idle=nomwait disable.ipv6=1 ghes.disable=1 printk.devkmsg=on"
$ sudo grub-mkconfig -o /boot/grub/grub.cfg
$ reboot
# check the acpi_idle settings and also the C state handling of the AMD Epyc:
$ cpupower idle-info

Seems to make things better, but does not realy solve the problem

Microcode Upgrade via Linux Kernel on Boot

Kernel upgrade doesnt upgrade AMD Firmware properly:

$ dmesg | grep 'microcode'
# ...
# x86/CPU: CPU features have changed after loading microcode, but might not take effect

so do it yourself

~ $ mkdir -p kernel/x86/microcode
# mkdir: created directory 'kernel'
# mkdir: created directory 'kernel/x86'
# mkdir: created directory 'kernel/x86/microcode'
~ $ cat /lib/firmware/amd-ucode/microcode_amd*.bin > kernel/x86/microcode/AuthenticAMD.bin
~ $ echo kernel/x86/microcode/AuthenticAMD.bin | bsdcpio -o -H newc -R 0:0 > amd-ucode.img
# 50 blocks
~ $ sudo mv amd-ucode.img /boot/.
~ $ sudo grub-mkconfig -o /boot/grub/grub.cfg
# Generating grub configuration file ...
# Found linux image: /boot/vmlinuz-linux
# Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux.img
# Found fallback initrd image(s) in /boot: initramfs-linux-fallback.img
# done

and it works

~ $ reboot
~ $ dmesg | grep microcode
# 1381:[  +0.000068] microcode: Microcode Update Driver: v2.2.

See also

Archlinux amd-ucode package (since 2018-08-26)

$ pac amd-ucode
# grub-mkconfig will automatically detect the microcode update and configure GRUB appropriately.
$ grub-mkconfig -o /boot/grub/grub.cfg

Disabling late microcode updates

“…For AMD systems the CPU microcode will get updated even if amd-ucode in not installed since the files are provided by linux-firmware (FS#59840). To disable late loading you must override the tmpfile /usr/lib/tmpfiles.d/linux-firmware.conf. It can be done by creating a file with the same filename in /etc/tmpfiles.d/:…”

$ ln -s /dev/null /etc/tmpfiles.d/linux-firmware.conf
$ reboot

Verifying that microcode got updated on boot

$ dmesg | grep microcode
wiki/amd-epyc/microcode-update-on-boot.txt · Last modified: 2021/02/25 11:26 by schild