[TRACKING] Kworker stuck at near 100% CPU usage with Ubuntu 22.04

Any comments on the above, @Matt_Hartley ? The issue ought to be reopened. It is not solved.

Okay, so we have devices attached. I will match the devices attached (this time also adding the Ethernet expansion card to see if I can replicate this and call it a bug).

If I cannot, then we may have a hardware issue with either the expansion card or the board itself.

As I prepare an external USB pendrive to test the booting from it as you suggested: any linux OS you’d prefer? Ubuntu 22.04 LTS would do?

Is a good choice. Also Fedora 37.

Booting from a USB drive gives the same result: after plugging in the ethernet expansion card, the kworker jumps to ~90% CPU usage.

I’ve traced what the kworker is doing with:

# trace-cmd record -e workqueue:workqueue_queue_work
# trace-cmd report > trace.log

And these are the first 100 lines:

cpus=8
       trace-cmd-809966 [003] 69662.878204: workqueue_queue_work: work struct=0xffff988679f49c08 function=flush_to_ldisc workqueue=events_unbound req_cpu=8192 cpu=4294967295
     kworker/4:1-791872 [004] 69662.878226: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878235: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
            sudo-809964 [002] 69662.878241: workqueue_queue_work: work struct=0xffff98860a3f2408 function=flush_to_ldisc workqueue=events_unbound req_cpu=8192 cpu=4294967295
     kworker/4:1-791872 [004] 69662.878290: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878298: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878353: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878361: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878416: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878424: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878478: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878486: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878541: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878549: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878603: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878611: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878666: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878674: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878729: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878737: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878791: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878799: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878854: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878862: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878917: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878925: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878979: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.878987: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879042: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879050: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879105: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879113: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879167: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879175: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879230: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879238: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879293: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879301: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879355: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879363: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879401: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879409: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879463: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879471: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879526: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879534: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879588: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879596: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879651: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879659: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879713: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879721: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879775: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879783: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879838: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879846: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879884: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879891: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879929: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879937: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.879992: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880000: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880055: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880063: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880101: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880108: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880163: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880171: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880226: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880234: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880289: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880297: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880351: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880359: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880414: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880422: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880477: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880485: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880523: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880531: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880585: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880616: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880670: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880678: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880733: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880741: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880795: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880803: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880887: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880895: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880949: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.880957: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.881012: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.881020: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.881075: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.881083: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.881137: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.881145: workqueue_queue_work: work struct=0xffff988aec840218 function=pm_runtime_work workqueue=pm req_cpu=8192 cpu=4
     kworker/4:1-791872 [004] 69662.881183: workqueue_queue_work: work struct=0xffff988c9c9e7dc0 function=hub_event workqueue=usb_hub_wq req_cpu=8192 cpu=4

Only “solution” I know is sudo modprobe -r xhci-pci, but of course then the buiilt-in camera and the ethernet expansion card don’t work.

To confirm:

  • Booted to Live USB only, not booted to the hard drive.

  • Inserted Ethernet expansion card, CPU spike?

This is beginning to feel like a bad expansion card as it cannot replicate it here with a known to be good working card.

Try one more time, expansion card inserted powered off. Then boot to the Live USB, report back.