Device stopped for over a day and needs to be restarted daily
Device stopped for over a day and needs to be restarted daily
Consider these options: 1. Look for kernel issues or errors before freezes. If your system stops responding completely, start a persistent log using journalctl and monitor it after each crash. Also, search crash_log.txt from another device or recovery mode. Checking for kernel panics can help identify what causes the freeze. 2. Examine SSD condition. Even if everything works normally before freezing, the SSD might be deteriorating without SMART detection. 3. Test RAM performance at various speeds. Although MemTest passed, older Ryzen 1st-gen CPUs may react poorly to different RAM rates. Adjust RAM speed in BIOS (e.g., lower from 3200MHz to 2666MHz) and disable XMP/DOCP if using them. 4. Switch to a different kernel version. If the problem began after a recent Ubuntu update, the new kernel could be incompatible with your hardware. 5. Verify stability without Docker. If Docker containers are the source of instability, try running the system without them. Stop all containers and keep the machine active to see if crashes occur. If stable, a Docker container might be the culprit. 6. Investigate PCIe connections (GPU or SSD). Since the system freezes completely, a PCIe problem is likely. Look for bus resets or timeout alerts. Move the GPU to another PCIe slot if feasible. Ensure the SSD is correctly seated in its designated port. 7. Replace or boot from an alternative drive. If you have a spare HDD/SSD, install Ubuntu on it and observe if the system remains functional. A failing SSD would likely cause crashes in this scenario.
ystemd state
stack:0 pid:1 tgid:1 ppid:0 flags:0x00000002 Mar 03 01:02:17 immich-server.local kernel: Call Trace: Mar 03 01:02:17 immich-server.local kernel: <TASK> Mar 03 01:02:17 immich-server.local kernel: __schedule+0x27c/0x6b0 Mar 03 01:02:17 immich-server.local kernel: schedule+0x33/0x110 Mar 03 01:02:17 immich-server.local kernel: schedule_timeout+0x157/0x170 Mar 03 01:02:17 immich-server.local kernel: wait_for_completion+0x88/0x150 Mar 03 01:02:17 immich-server.local kernel: __wait_rcu_gp+0x17e/0x190 Mar 03 01:02:17 immich-server.local kernel: synchronize_rcu+0x12d/0x140 Mar 03 01:02:17 immich-server.local kernel: ? __pfx_call_rcu_hurry+0x10/0x10 Mar 03 01:02:17 immich-server.local kernel: ? __pfx_wakeme_after_rcu+0x10/0x10 Mar 03 01:02:17 immich-server.local kernel: rcu_sync_enter+0x59/0x120 Mar 03 01:02:17 immich-server.local kernel: ? _kstrtoull+0x3b/0xa0 Mar 03 01:02:17 immich-server.local kernel: percpu_down_write+0x2b/0x1d0 Mar 03 01:02:17 immich-server.local kernel: cgroup_procs_write_start+0x116/0x190 Mar 03 01:02:17 immich-server.local kernel: __cgroup_procs_write+0x5f/0x170 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: cgroup_procs_write+0x17/0x30 Mar 03 01:02:17 immich-server.local kernel: cgroup_file_write+0x8d/0x1b0 Mar 03 01:02:17 immich-server.local kernel: ? __check_object_size.part.0+0x72/0x150 Mar 03 01:02:17 immich-server.local kernel: kernfs_fop_write_iter+0x14f/0x1e0 Mar 03 01:02:17 immich-server.local kernel: vfs_write+0x2a8/0x480 Mar 03 01:02:17 immich-server.local kernel: ksys_write+0x73/0x100 Mar 03 01:02:17 immich-server.local kernel: __x64_sys_write+0x19/0x30 Mar 03 01:02:17 immich-server.local kernel: x64_sys_call+0x7e/0x25a0 Mar 03 01:02:17 immich-server.local kernel: do_syscall_64+0x7f/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_sys_openat2+0x9f/0xe0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __x64_sys_openat+0x55/0xa0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __x64_sys_fcntl+0xba/0xe0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __set_task_blocked+0x29/0x80 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? sigprocmask+0x8c/0xd0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __x64_sys_rt_sigprocmask+0x7f/0xe0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? mntput+0x24/0x50 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? path_put+0x1e/0x30 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? proc_pid_readlink+0x167/0x1e0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? mntput_no_expire+0x51/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __set_task_blocked+0x29/0x80 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? sigprocmask+0x8c/0xd0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? _copy_to_user+0x25/0x50 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __x64_sys_rt_sigprocmask+0x99/0xe0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? irqentry_exit+0x43/0x50 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? exc_page_fault+0x94/0x1b0 Mar 03 01:02:17 immich-server.local kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80 Mar 03 01:02:17 immich-server.local kernel: RIP: 0033:0x7345b091c574 Mar 03 01:02:17 immich-server.local kernel: RSP: 002b:00007fffa01cecf8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 Mar 03 01:02:17 immich-server.local kernel: RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00007345b091c574 Mar 03 01:02:17 immich-server.local kernel: RDX: 0000000000000006 RSI: 00007fffa01ceeca RDI: 0000000000000052 Mar 03 01:02:17 immich-server.local kernel: RBP: 00007fffa01ced20 R08: 00007345b0a02228 R09: 0000000000000000 Mar 03 01:02:17 immich-server.local kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000006 Mar 03 01:02:17 immich-server.local kernel: R13: 00007fffa01ceeca R14: 00005657cd03f510 R15: 0000000000000006 Mar 03 01:02:17 immich-server.local kernel: </TASK> Mar 03 01:02:17 immich-server.local kernel: INFO: task systemd-udevd:609 blocked for more than 122 seconds. Mar 03 01:02:17 immich-server.local kernel: Tainted: P O 6.8.0-54-generic #56-Ubuntu Mar 03 01:02:17 immich-server.local kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 03 01:02:17 immich-server.local kernel: taskystemd-udevd state stack:0 pid:609 tgid:609 ppid:1 flags:0x00004002 Mar 03 01:02:17 immich-server.local kernel: Call Trace: Mar 03 01:02:17 immich-server.local kernel: <TASK> Mar 03 01:02:17 immich-server.local kernel: __schedule+0x27c/0x6b0 Mar 03 01:02:17 immich-server.local kernel: schedule+0x33/0x110 Mar 03 01:02:17 immich-server.local kernel: schedule_timeout+0x157/0x170 Mar 03 01:02:17 immich-server.local kernel: wait_for_completion_state+0x146/0x1f0 Mar 03 01:02:17 immich-server.local kernel: call_usermodehelper_exec+0x183/0x1c0 Mar 03 01:02:17 immich-server.local kernel: do_coredump+0x3d6/0xb30 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? kmem_cache_free+0x3cd/0x480 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? signalfd_poll+0x87/0xc0 Mar 03 01:02:17 immich-server.local kernel: get_signal+0x84f/0x9b0 Mar 03 01:02:17 immich-server.local kernel: arch_do_signal_or_restart+0x39/0x120 Mar 03 01:02:17 immich-server.local kernel: syscall_exit_to_user_mode+0x206/0x260 Mar 03 01:02:17 immich-server.local kernel: do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __memcg_slab_free_hook+0x115/0x180 Mar 03 01:02:17 immich-server.local kernel: ? __fput+0x15e/0x2e0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? kmem_cache_free+0x458/0x480 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __fput+0x15e/0x2e0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __fput_sync+0x1c/0x30 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? exc_page_fault+0x94/0x1b0 Mar 03 01:02:17 immich-server.local kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80 Mar 03 01:02:17 immich-server.local kernel: RIP: 0033:0x75f98512a007 Mar 03 01:02:17 immich-server.local kernel: RSP: 002b:00007fff2b557fc8 EFLAGS: 00000202 ORIG_RAX: 00000000000000e8 Mar 03 01:02:17 immich-server.local kernel: RAX: fffffffffffffffc RBX: 000000000000000b RCX: 000075f98512a007 Mar 03 01:02:17 immich-server.local kernel: RDX: 0000000000000134 RSI: 00005593261246d0 RDI: 0000000000000009 Mar 03 01:02:17 immich-server.local kernel: RBP: 00007fff2b5580e0 R08: 00007fff2b557e30 R09: 0000000000000001 Mar 03 01:02:17 immich-server.local kernel: R10: 00000000ffffffff R11: 0000000000000202 R12: 000000000000006e Mar 03 01:02:17 immich-server.local kernel: R13: 0000559325fdf2d8 R14: 0000000000000134 R15: 0000559325fdf140 Mar 03 01:02:17 immich-server.local kernel: </TASK> Mar 03 01:02:17 immich-server.local kernel: INFO: task (sa1):19358 blocked for more than 122 seconds. Mar 03 01:02:17 immich-server.local kernel: Tainted: P O 6.8.0-54-generic #56-Ubuntu Mar 03 01:02:17 immich-server.local kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 03 01:02:17 immich-server.local kernel: task
sa1) state stack:0 pid:19358 tgid:19358 ppid:1 flags:0x00000002 Mar 03 01:02:17 immich-server.local kernel: Call Trace: Mar 03 01:02:17 immich-server.local kernel: <TASK> Mar 03 01:02:17 immich-server.local kernel: __schedule+0x27c/0x6b0 Mar 03 01:02:17 immich-server.local kernel: schedule+0x33/0x110 Mar 03 01:02:17 immich-server.local kernel: schedule_preempt_disabled+0x15/0x30 Mar 03 01:02:17 immich-server.local kernel: __mutex_lock.constprop.0+0x42f/0x740 Mar 03 01:02:17 immich-server.local kernel: __mutex_lock_slowpath+0x13/0x20 Mar 03 01:02:17 immich-server.local kernel: mutex_lock+0x3c/0x50 Mar 03 01:02:17 immich-server.local kernel: cgroup_kn_lock_live+0x4a/0xf0 Mar 03 01:02:17 immich-server.local kernel: __cgroup_procs_write+0x40/0x170 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: cgroup_procs_write+0x17/0x30 Mar 03 01:02:17 immich-server.local kernel: cgroup_file_write+0x8d/0x1b0 Mar 03 01:02:17 immich-server.local kernel: ? __check_object_size.part.0+0x72/0x150 Mar 03 01:02:17 immich-server.local kernel: kernfs_fop_write_iter+0x14f/0x1e0 Mar 03 01:02:17 immich-server.local kernel: vfs_write+0x2a8/0x480 Mar 03 01:02:17 immich-server.local kernel: ksys_write+0x73/0x100 Mar 03 01:02:17 immich-server.local kernel: __x64_sys_write+0x19/0x30 Mar 03 01:02:17 immich-server.local kernel: x64_sys_call+0x7e/0x25a0 Mar 03 01:02:17 immich-server.local kernel: do_syscall_64+0x7f/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __do_sys_statfs+0x72/0x80 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? xas_find+0x74/0x1e0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? next_uptodate_folio+0xa9/0x320 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? filemap_map_pages+0x2fe/0x4c0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_read_fault+0x112/0x200 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_fault+0xf0/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? handle_pte_fault+0x114/0x1d0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __handle_mm_fault+0x654/0x800 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __count_memcg_events+0x6b/0x120 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? count_memcg_events.constprop.0+0x2a/0x50 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? handle_mm_fault+0xad/0x380 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_user_addr_fault+0x333/0x670 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? irqentry_exit_to_user_mode+0x7b/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? irqentry_exit+0x43/0x50 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? exc_page_fault+0x94/0x1b0 Mar 03 01:02:17 immich-server.local kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80 Mar 03 01:02:17 immich-server.local kernel: RIP: 0033:0x78ab8e51c574 Mar 03 01:02:17 immich-server.local kernel: RSP: 002b:00007ffec07e2dd8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 Mar 03 01:02:17 immich-server.local kernel: RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 000078ab8e51c574 Mar 03 01:02:17 immich-server.local kernel: RDX: 0000000000000006 RSI: 00007ffec07e2faa RDI: 0000000000000003 Mar 03 01:02:17 immich-server.local kernel: RBP: 00007ffec07e2e00 R08: 0000000000000000 R09: 0000000000000000 Mar 03 01:02:17 immich-server.local kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000006 Mar 03 01:02:17 immich-server.local kernel: R13: 00007ffec07e2faa R14: 0000598c3f8692d0 R15: 0000000000000006 Mar 03 01:02:17 immich-server.local kernel: </TASK> Mar 03 01:04:06 immich-server.local kernel: watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [python3:19061] Mar 03 01:04:06 immich-server.local kernel: Modules linked in: xt_nat xt_tcpudp nf_conntrack_netlink veth xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bridge stp llc xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables qrtr overlay binfmt_misc amdgpu intel_rapl_msr intel_rapl_common edac_mce_amd snd_hda_codec_realtek ath10k_pci snd_hda_codec_generic snd_hda_codec_hdmi amdxcp ath10k_core drm_exec gpu_sched snd_hda_intel kvm_amd drm_buddy snd_intel_dspcfg ath drm_suballoc_helper snd_intel_sdw_acpi nls_iso8859_1 snd_hda_codec zfs(PO) drm_ttm_helper mac80211 snd_hda_core ttm btusb kvm btrtl snd_hwdep drm_display_helper btintel dell_wmi snd_pcm btbcm cec btmtk dell_smbios snd_timer dell_smm_hwmon spl(O) dcdbas irqbypass rc_core i2c_algo_bit ledtrig_audio bluetooth cfg80211 alienware_wmi video snd sparse_keymap rapl wmi_bmof dell_wmi_descriptor serio_raw ecdh_generic soundcore libarc4 k1 I'll be honest, I'm not too sure why it happened so I asked AI. Possible Causes: Firmware Update (fwupd) Triggering Hardware Issues Hardware or Driver Conflict Cgroup Lock Contention Also found this: immich-server:~$ sudo dmesg -T | grep -iE "error|fail|firmware" [Mon Mar 3 01:20:16 2025] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored [Mon Mar 3 01:20:16 2025] ACPI: _OSC evaluation for CPUs failed, trying _PDC [Mon Mar 3 01:20:16 2025] acpi PNP0A08:00: [Firmware Info]: ECAM [mem 0xf8000000-0xfbffffff] for domain 0000 [bus 00-3f] only partially covers this bridge [Mon Mar 3 01:20:17 2025] AMD-Vi: [Firmware Warn]: EFR mismatch. Use IVHD EFR (0xf77ef22294ada : 0x400f77ef22294ada), EFR2 (0x0 : 0x0). [Mon Mar 3 01:20:17 2025] RAS: Correctable Errors collector initialized. [Mon Mar 3 01:20:23 2025] ath10k_pci 0000:05:00.0: firmware ver WLAN.RM.4.4.1-00309- api 6 features wowlan,ignore-otp,mfp crc32 0793bcf2 [Mon Mar 3 01:20:29 2025] [drm] Loading DMUB firmware via PSP: version=0x02020020 [Mon Mar 3 01:20:29 2025] [drm] Found VCN firmware Version ENC: 1.30 DEC: 3 VEP: 0 Revision: 4 [Mon Mar 3 01:20:29 2025] amdgpu 0000:0b:00.0: amdgpu: Will use PSP to load VCN firmware Going to disable wifi since im on ethernet anyways. How might I check if hardware/drivers are at cause? For now, I've disabled fwupd and will continue monitoring the system to see if hangs again.
Amegahoney
11-10-2022, 01:08 PM #13thanks for this! I was able to capture where the kernel locked up: Mar 03 00:57:19 immich-server.local systemd[1]: Starting fwupd-refresh.service - Refresh fwupd metadata and update motd... Mar 03 00:57:19 immich-server.local dbus-daemon[1313]: [system] Activating via systemd: service name='org.freedesktop.fwupd' unit='fwupd.service' requested by ':1.71' (uid=989 pid=19256 comm="/usr/bin/fwupdmgr refresh" label="unconfined") Mar 03 00:57:19 immich-server.local systemd[1]: Starting fwupd.service - Firmware update daemon... Mar 03 00:57:19 immich-server.local dbus-daemon[1313]: [system] Activating via systemd: service name='org.freedesktop.UPower' unit='upower.service' requested by ':1.72' (uid=0 pid=19261 comm="/usr/libexec/fwupd/fwupd" label="unconfined") Mar 03 00:57:19 immich-server.local systemd[1]: Starting upower.service - Daemon for power management... Mar 03 00:57:19 immich-server.local dbus-daemon[1313]: [system] Successfully activated service 'org.freedesktop.UPower' Mar 03 00:57:19 immich-server.local systemd[1]: Started upower.service - Daemon for power management. Mar 03 00:57:20 immich-server.local fwupd[19261]: 00:57:20.030 FuMain Daemon ready for requests (locale en_US.UTF-8) Mar 03 00:57:20 immich-server.local dbus-daemon[1313]: [system] Successfully activated service 'org.freedesktop.fwupd' Mar 03 00:57:20 immich-server.local systemd[1]: Started fwupd.service - Firmware update daemon. Mar 03 00:57:20 immich-server.local fwupdmgr[19256]: Updating lvfs Mar 03 00:57:20 immich-server.local fwupdmgr[19256]: Successfully downloaded new metadata: 1 local device supported Mar 03 00:57:20 immich-server.local systemd[1]: fwupd-refresh.service: Deactivated successfully. Mar 03 00:57:20 immich-server.local systemd[1]: Finished fwupd-refresh.service - Refresh fwupd metadata and update motd. Mar 03 00:59:19 immich-server.local systemd[1]: systemd-udevd.service: Watchdog timeout (limit 3min)! Mar 03 00:59:19 immich-server.local systemd[1]: systemd-udevd.service: Killing process 609 (systemd-udevd) with signal SIGABRT. Mar 03 01:02:17 immich-server.local kernel: INFO: task systemd:1 blocked for more than 122 seconds. Mar 03 01:02:17 immich-server.local kernel: Tainted: P O 6.8.0-54-generic #56-Ubuntu Mar 03 01:02:17 immich-server.local kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 03 01:02:17 immich-server.local kernel: taskystemd state stack:0 pid:1 tgid:1 ppid:0 flags:0x00000002 Mar 03 01:02:17 immich-server.local kernel: Call Trace: Mar 03 01:02:17 immich-server.local kernel: <TASK> Mar 03 01:02:17 immich-server.local kernel: __schedule+0x27c/0x6b0 Mar 03 01:02:17 immich-server.local kernel: schedule+0x33/0x110 Mar 03 01:02:17 immich-server.local kernel: schedule_timeout+0x157/0x170 Mar 03 01:02:17 immich-server.local kernel: wait_for_completion+0x88/0x150 Mar 03 01:02:17 immich-server.local kernel: __wait_rcu_gp+0x17e/0x190 Mar 03 01:02:17 immich-server.local kernel: synchronize_rcu+0x12d/0x140 Mar 03 01:02:17 immich-server.local kernel: ? __pfx_call_rcu_hurry+0x10/0x10 Mar 03 01:02:17 immich-server.local kernel: ? __pfx_wakeme_after_rcu+0x10/0x10 Mar 03 01:02:17 immich-server.local kernel: rcu_sync_enter+0x59/0x120 Mar 03 01:02:17 immich-server.local kernel: ? _kstrtoull+0x3b/0xa0 Mar 03 01:02:17 immich-server.local kernel: percpu_down_write+0x2b/0x1d0 Mar 03 01:02:17 immich-server.local kernel: cgroup_procs_write_start+0x116/0x190 Mar 03 01:02:17 immich-server.local kernel: __cgroup_procs_write+0x5f/0x170 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: cgroup_procs_write+0x17/0x30 Mar 03 01:02:17 immich-server.local kernel: cgroup_file_write+0x8d/0x1b0 Mar 03 01:02:17 immich-server.local kernel: ? __check_object_size.part.0+0x72/0x150 Mar 03 01:02:17 immich-server.local kernel: kernfs_fop_write_iter+0x14f/0x1e0 Mar 03 01:02:17 immich-server.local kernel: vfs_write+0x2a8/0x480 Mar 03 01:02:17 immich-server.local kernel: ksys_write+0x73/0x100 Mar 03 01:02:17 immich-server.local kernel: __x64_sys_write+0x19/0x30 Mar 03 01:02:17 immich-server.local kernel: x64_sys_call+0x7e/0x25a0 Mar 03 01:02:17 immich-server.local kernel: do_syscall_64+0x7f/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_sys_openat2+0x9f/0xe0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __x64_sys_openat+0x55/0xa0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __x64_sys_fcntl+0xba/0xe0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __set_task_blocked+0x29/0x80 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? sigprocmask+0x8c/0xd0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __x64_sys_rt_sigprocmask+0x7f/0xe0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? mntput+0x24/0x50 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? path_put+0x1e/0x30 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? proc_pid_readlink+0x167/0x1e0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? mntput_no_expire+0x51/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __set_task_blocked+0x29/0x80 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? sigprocmask+0x8c/0xd0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? _copy_to_user+0x25/0x50 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __x64_sys_rt_sigprocmask+0x99/0xe0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? irqentry_exit+0x43/0x50 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? exc_page_fault+0x94/0x1b0 Mar 03 01:02:17 immich-server.local kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80 Mar 03 01:02:17 immich-server.local kernel: RIP: 0033:0x7345b091c574 Mar 03 01:02:17 immich-server.local kernel: RSP: 002b:00007fffa01cecf8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 Mar 03 01:02:17 immich-server.local kernel: RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00007345b091c574 Mar 03 01:02:17 immich-server.local kernel: RDX: 0000000000000006 RSI: 00007fffa01ceeca RDI: 0000000000000052 Mar 03 01:02:17 immich-server.local kernel: RBP: 00007fffa01ced20 R08: 00007345b0a02228 R09: 0000000000000000 Mar 03 01:02:17 immich-server.local kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000006 Mar 03 01:02:17 immich-server.local kernel: R13: 00007fffa01ceeca R14: 00005657cd03f510 R15: 0000000000000006 Mar 03 01:02:17 immich-server.local kernel: </TASK> Mar 03 01:02:17 immich-server.local kernel: INFO: task systemd-udevd:609 blocked for more than 122 seconds. Mar 03 01:02:17 immich-server.local kernel: Tainted: P O 6.8.0-54-generic #56-Ubuntu Mar 03 01:02:17 immich-server.local kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 03 01:02:17 immich-server.local kernel: taskystemd-udevd state stack:0 pid:609 tgid:609 ppid:1 flags:0x00004002 Mar 03 01:02:17 immich-server.local kernel: Call Trace: Mar 03 01:02:17 immich-server.local kernel: <TASK> Mar 03 01:02:17 immich-server.local kernel: __schedule+0x27c/0x6b0 Mar 03 01:02:17 immich-server.local kernel: schedule+0x33/0x110 Mar 03 01:02:17 immich-server.local kernel: schedule_timeout+0x157/0x170 Mar 03 01:02:17 immich-server.local kernel: wait_for_completion_state+0x146/0x1f0 Mar 03 01:02:17 immich-server.local kernel: call_usermodehelper_exec+0x183/0x1c0 Mar 03 01:02:17 immich-server.local kernel: do_coredump+0x3d6/0xb30 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? kmem_cache_free+0x3cd/0x480 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? signalfd_poll+0x87/0xc0 Mar 03 01:02:17 immich-server.local kernel: get_signal+0x84f/0x9b0 Mar 03 01:02:17 immich-server.local kernel: arch_do_signal_or_restart+0x39/0x120 Mar 03 01:02:17 immich-server.local kernel: syscall_exit_to_user_mode+0x206/0x260 Mar 03 01:02:17 immich-server.local kernel: do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __memcg_slab_free_hook+0x115/0x180 Mar 03 01:02:17 immich-server.local kernel: ? __fput+0x15e/0x2e0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? kmem_cache_free+0x458/0x480 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __fput+0x15e/0x2e0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __fput_sync+0x1c/0x30 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? syscall_exit_to_user_mode+0x86/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_syscall_64+0x8c/0x180 Mar 03 01:02:17 immich-server.local kernel: ? exc_page_fault+0x94/0x1b0 Mar 03 01:02:17 immich-server.local kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80 Mar 03 01:02:17 immich-server.local kernel: RIP: 0033:0x75f98512a007 Mar 03 01:02:17 immich-server.local kernel: RSP: 002b:00007fff2b557fc8 EFLAGS: 00000202 ORIG_RAX: 00000000000000e8 Mar 03 01:02:17 immich-server.local kernel: RAX: fffffffffffffffc RBX: 000000000000000b RCX: 000075f98512a007 Mar 03 01:02:17 immich-server.local kernel: RDX: 0000000000000134 RSI: 00005593261246d0 RDI: 0000000000000009 Mar 03 01:02:17 immich-server.local kernel: RBP: 00007fff2b5580e0 R08: 00007fff2b557e30 R09: 0000000000000001 Mar 03 01:02:17 immich-server.local kernel: R10: 00000000ffffffff R11: 0000000000000202 R12: 000000000000006e Mar 03 01:02:17 immich-server.local kernel: R13: 0000559325fdf2d8 R14: 0000000000000134 R15: 0000559325fdf140 Mar 03 01:02:17 immich-server.local kernel: </TASK> Mar 03 01:02:17 immich-server.local kernel: INFO: task (sa1):19358 blocked for more than 122 seconds. Mar 03 01:02:17 immich-server.local kernel: Tainted: P O 6.8.0-54-generic #56-Ubuntu Mar 03 01:02:17 immich-server.local kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 03 01:02:17 immich-server.local kernel: tasksa1) state stack:0 pid:19358 tgid:19358 ppid:1 flags:0x00000002 Mar 03 01:02:17 immich-server.local kernel: Call Trace: Mar 03 01:02:17 immich-server.local kernel: <TASK> Mar 03 01:02:17 immich-server.local kernel: __schedule+0x27c/0x6b0 Mar 03 01:02:17 immich-server.local kernel: schedule+0x33/0x110 Mar 03 01:02:17 immich-server.local kernel: schedule_preempt_disabled+0x15/0x30 Mar 03 01:02:17 immich-server.local kernel: __mutex_lock.constprop.0+0x42f/0x740 Mar 03 01:02:17 immich-server.local kernel: __mutex_lock_slowpath+0x13/0x20 Mar 03 01:02:17 immich-server.local kernel: mutex_lock+0x3c/0x50 Mar 03 01:02:17 immich-server.local kernel: cgroup_kn_lock_live+0x4a/0xf0 Mar 03 01:02:17 immich-server.local kernel: __cgroup_procs_write+0x40/0x170 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: cgroup_procs_write+0x17/0x30 Mar 03 01:02:17 immich-server.local kernel: cgroup_file_write+0x8d/0x1b0 Mar 03 01:02:17 immich-server.local kernel: ? __check_object_size.part.0+0x72/0x150 Mar 03 01:02:17 immich-server.local kernel: kernfs_fop_write_iter+0x14f/0x1e0 Mar 03 01:02:17 immich-server.local kernel: vfs_write+0x2a8/0x480 Mar 03 01:02:17 immich-server.local kernel: ksys_write+0x73/0x100 Mar 03 01:02:17 immich-server.local kernel: __x64_sys_write+0x19/0x30 Mar 03 01:02:17 immich-server.local kernel: x64_sys_call+0x7e/0x25a0 Mar 03 01:02:17 immich-server.local kernel: do_syscall_64+0x7f/0x180 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __do_sys_statfs+0x72/0x80 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? xas_find+0x74/0x1e0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? next_uptodate_folio+0xa9/0x320 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? filemap_map_pages+0x2fe/0x4c0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_read_fault+0x112/0x200 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_fault+0xf0/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? handle_pte_fault+0x114/0x1d0 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __handle_mm_fault+0x654/0x800 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? __count_memcg_events+0x6b/0x120 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? count_memcg_events.constprop.0+0x2a/0x50 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? handle_mm_fault+0xad/0x380 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? do_user_addr_fault+0x333/0x670 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? irqentry_exit_to_user_mode+0x7b/0x260 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? irqentry_exit+0x43/0x50 Mar 03 01:02:17 immich-server.local kernel: ? srso_return_thunk+0x5/0x5f Mar 03 01:02:17 immich-server.local kernel: ? exc_page_fault+0x94/0x1b0 Mar 03 01:02:17 immich-server.local kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80 Mar 03 01:02:17 immich-server.local kernel: RIP: 0033:0x78ab8e51c574 Mar 03 01:02:17 immich-server.local kernel: RSP: 002b:00007ffec07e2dd8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 Mar 03 01:02:17 immich-server.local kernel: RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 000078ab8e51c574 Mar 03 01:02:17 immich-server.local kernel: RDX: 0000000000000006 RSI: 00007ffec07e2faa RDI: 0000000000000003 Mar 03 01:02:17 immich-server.local kernel: RBP: 00007ffec07e2e00 R08: 0000000000000000 R09: 0000000000000000 Mar 03 01:02:17 immich-server.local kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000006 Mar 03 01:02:17 immich-server.local kernel: R13: 00007ffec07e2faa R14: 0000598c3f8692d0 R15: 0000000000000006 Mar 03 01:02:17 immich-server.local kernel: </TASK> Mar 03 01:04:06 immich-server.local kernel: watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [python3:19061] Mar 03 01:04:06 immich-server.local kernel: Modules linked in: xt_nat xt_tcpudp nf_conntrack_netlink veth xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bridge stp llc xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables qrtr overlay binfmt_misc amdgpu intel_rapl_msr intel_rapl_common edac_mce_amd snd_hda_codec_realtek ath10k_pci snd_hda_codec_generic snd_hda_codec_hdmi amdxcp ath10k_core drm_exec gpu_sched snd_hda_intel kvm_amd drm_buddy snd_intel_dspcfg ath drm_suballoc_helper snd_intel_sdw_acpi nls_iso8859_1 snd_hda_codec zfs(PO) drm_ttm_helper mac80211 snd_hda_core ttm btusb kvm btrtl snd_hwdep drm_display_helper btintel dell_wmi snd_pcm btbcm cec btmtk dell_smbios snd_timer dell_smm_hwmon spl(O) dcdbas irqbypass rc_core i2c_algo_bit ledtrig_audio bluetooth cfg80211 alienware_wmi video snd sparse_keymap rapl wmi_bmof dell_wmi_descriptor serio_raw ecdh_generic soundcore libarc4 k1 I'll be honest, I'm not too sure why it happened so I asked AI. Possible Causes: Firmware Update (fwupd) Triggering Hardware Issues Hardware or Driver Conflict Cgroup Lock Contention Also found this: immich-server:~$ sudo dmesg -T | grep -iE "error|fail|firmware" [Mon Mar 3 01:20:16 2025] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored [Mon Mar 3 01:20:16 2025] ACPI: _OSC evaluation for CPUs failed, trying _PDC [Mon Mar 3 01:20:16 2025] acpi PNP0A08:00: [Firmware Info]: ECAM [mem 0xf8000000-0xfbffffff] for domain 0000 [bus 00-3f] only partially covers this bridge [Mon Mar 3 01:20:17 2025] AMD-Vi: [Firmware Warn]: EFR mismatch. Use IVHD EFR (0xf77ef22294ada : 0x400f77ef22294ada), EFR2 (0x0 : 0x0). [Mon Mar 3 01:20:17 2025] RAS: Correctable Errors collector initialized. [Mon Mar 3 01:20:23 2025] ath10k_pci 0000:05:00.0: firmware ver WLAN.RM.4.4.1-00309- api 6 features wowlan,ignore-otp,mfp crc32 0793bcf2 [Mon Mar 3 01:20:29 2025] [drm] Loading DMUB firmware via PSP: version=0x02020020 [Mon Mar 3 01:20:29 2025] [drm] Found VCN firmware Version ENC: 1.30 DEC: 3 VEP: 0 Revision: 4 [Mon Mar 3 01:20:29 2025] amdgpu 0000:0b:00.0: amdgpu: Will use PSP to load VCN firmware Going to disable wifi since im on ethernet anyways. How might I check if hardware/drivers are at cause? For now, I've disabled fwupd and will continue monitoring the system to see if hangs again.