Laptop powers *on* after hibernate when unplugging/unpowering dock

Community Support
1

I’m having a peculiar issue with my setup. I have a Caldigit TS4 dock connected (which also powers the laptop). At the end of the day, my routine is:

  • Hibernate (suspend-to-disk) the laptop (systemctl hibernate)
  • Wait until it is really off
  • Power off the dock (I’m using a switched outlet to cut 230V to the dock).
  • The laptop powers up again

Of course, having the laptop power up again in this situation is counterproductive, and I’d like to avoid it.

Here’s some things I tested:

  • This issue occurs even when “Power up on AC” (or whatever it’s called exactly) in the BIOS is disabled, so seems to be unrelated to that.
  • This issue also occurs when I unplug the TB cable instead of powering down the dock, so I think this mean it cannot really be specific to this particular dock (at least it cannot be some message sent by the dock on powerdown).
  • This issue does not occur on a regular shutdown, so I think this means that when hibernating, the OS leaves some component in a different state than when powering down, maybe with instructions to power up again on some change.
  • This issue also occurs when nothing is attached to the dock, so it’s not that e.g. a USB keyboard is configured to wake up the laptop or so.
  • The issue does not occur with the Framework power adapter (so it’s not just that power is removed).
  • The issue does not occur with a plain powered USB2 hub connected (tried connecting it to a USB-A port and USB-C port using an adapter cable).
  • The issue does not occur when the dock is connected using USB3 or USB2 rather than TB (tested by inserting some non-TB-capable adapters into the uplink cable). This seems to suggest it is not caused by some USB device inside the dock, but it is the TB link itself, or maybe the PCIe-connected ethernet chip?
  • The issue still occurs when removing the igc kernel module (for the PCIe ethernet module in the dock).
  • The issue does not occur when removing the thunderbolt kernel module, so there is definitely something there. Also, as a workaround, I could remove this module before hibernate, maybe (does not seem to affect dock operation directly, so USB4 is still running without this module).

Does anyone recognize this behavior? Got any hints on where to look to fix this? I assume that the OS is involved somehow, but who ultimately decides to power on again? The EC? Or is there some other controller still running after hibernate? I’m not actually sure how hibernate works at all, even, whether this is a fully OS-managed suspend (user-space suspend?), or there is some ACPI-configured platform driver involved? Anyone know?

Edit: Added a few more test results above

1 Like
2

I just tested with my laptop, and while I don’t have a dock, hibernating my laptop and pulling out just a charger does not power on my laptop, so it seems to be something more than just USB-PD power causing your issue. I also tried with a power adapter plugged into a USB-C dongle, but still couldn’t get the same behavior you’re seeing. I wish I could offer more, but I hope you find a solution!

3

Thanks for testing! Your test prompted a few more tests of my own, I also tested with just the power adapter, with a powered USB2 hub, and with the dock connected through USB2 and USB3 rather than Thunderbolt, and an externally connected monitor using the DP expansion card (though that is DP alt-mode, while the dock uses DP tunneling), but none of these trigger the problem.

So I guess the issue is somehow related to either the TB connection itself, or the ethernet-over-PCIe module (wake-on-lan related maybe?).

So I tried removing the igc kernel module for the ethernet connection - problem still occurs.
Then I tried removing the thunderbolt module - problem stopped occurring!

So there’s definitely something in the kernel thunderbolt code here that I could be digging into. Interestingly enough, it seems all USB connections and even the DisplayPort output are unaffected by unloading the thunderbolt module, so I suspect that removing it will disable thunderbolt, but keep the USB4 connection (which is essentially what TB is, and over which USB3 and DP are tunneled) active (i.e. it does not fall back to USB3).

Here’s what dmesg says when unloading the thunderbolt module:

[36148.453408] thunderbolt 0-1: device disconnected
[36148.456654] ACPI: bus type thunderbolt unregistered

So apparently there is also some ACPI code involved here, maybe that sets up the wakeup source?

It also turns out that wakeup from standby does still work even without the thunderbolt module, so it’s not that either (but that does mean that simply blacklisting the thunderbolt module could be an effective workaround, seems it’s not actually needed for anything?).

I’ll have a look in the thunderbolt module options and source later, see if that shows anything relevant.

1 Like
4

One more observation: After hibernation, I can also turn on the system by pressing a key on the external keyboard. If the thunderbolt module is unloaded, this does not work. So this is probably the same mechanism that powers up the laptop on dock unplug/powerdown.

Looking at the thunderbolt module:

  • There do not seem to be any relevant module options
  • The module seems to be responsible for TB and USB4, so it is actually weird that USB4 remains working with the module unloaded. However, I suspect that the bulk of the USB4 operation might actually be handled by firmware running somewhere, which can handle tunneling of USB3, DP and PCIe traffic without needing to be controlled by the OS?

I tried enabling debug output for the thunderbolt module and acpi code using:

echo "module thunderbolt +p" > /sys/kernel/debug/dynamic_debug/contro
echo "file drivers/acpi/* +p" > /sys/kernel/debug/dynamic_debug/control

But that did not really offer any clues. For reference, here’s the log generated by a hibernate-and-resume cycle: Hibernate cycle on Framework laptop with Caldigit TS4 dock, with thunderbolt module loaded · GitHub (put on gist because of this forum’s post length limit).

With the thunderbolt module unloaded, it looks like this: Hibernate cycle on Framework laptop with Caldigit TS4 dock, without thunderbolt module loaded · GitHub

Comparing the hibernate parts (omitting the resume parts) of these (- == without thunderbolt, + == with thunderbolt) gives the below, but still does not seem to show clear wakeup causes being enabled…

--- dmesg-without-tb	2022-04-19 12:53:01.076143250 +0200
+++ dmesg-with-tb	2022-04-19 12:54:11.815421570 +0200
@@ -1,7 +1,10 @@
 ACPI: GPE event 0x66
 ACPI: GPE event 0x66
+ACPI: \_SB_.PC00.LPCB.EC0_.SEN3: ACPI: utils: Return value [3110]
+ACPI: \_SB_.PC00.LPCB.EC0_.SEN5: ACPI: utils: Return value [3130]
+ACPI: \_SB_.PC00.TCPU: ACPI: utils: Return value [3162]
 PM: hibernation: hibernation entry
-Filesystems sync: 0.015 seconds
+Filesystems sync: 0.012 seconds
 Freezing user space processes ... (elapsed 0.003 seconds) done.
 OOM killer disabled.
 PM: hibernation: Marking nosave pages: [mem 0x00000000-0x00000fff]
@@ -14,8 +17,8 @@
 PM: hibernation: Marking nosave pages: [mem 0x45c00000-0xffffffff]
 PM: hibernation: Basic memory bitmaps created
 PM: hibernation: Preallocating image memory
-PM: hibernation: Allocated 1537075 pages for snapshot
-PM: hibernation: Allocated 6148300 kbytes in 3.83 seconds (1605.30 MB/s)
+PM: hibernation: Allocated 1567347 pages for snapshot
+PM: hibernation: Allocated 6269388 kbytes in 3.11 seconds (2015.88 MB/s)
 Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
 ACPI: \_SB_.PC00.I2C0: ACPI: utils: Evaluate [_S4D]: AE_NOT_FOUND
 ACPI: \_SB_.PC00.I2C1: ACPI: utils: Evaluate [_S4D]: AE_NOT_FOUND
@@ -29,30 +32,55 @@
 pcieport 0000:02:00.0: Wakeup disabled by ACPI
 ACPI: \_SB_.PC00.I2C1: ACPI: PM: Power state change: D3hot -> D0
 acpi device:85: Power state changed to D0
-ACPI: \_SB_.PC00.I2C0: ACPI: PM: Power state change: D3hot -> D0
-acpi device:84: Power state changed to D0
-ACPI: \_SB_.PC00.TRP2: ACPI: PM: Power state change: D3cold -> D0
-ACPI: \_SB_.PC00.TRP3: ACPI: PM: Power state change: D3cold -> D0
+ACPI: \_SB_.PC00.TDM1: ACPI: PM: Power state change: D3cold -> D0
 ACPI: \_SB_.PC00.D3C_: ACPI: PM: Power resource already on
+ACPI: \_SB_.PC00.TRP3: ACPI: PM: Power state change: D3cold -> D0
+ACPI: \_SB_.PC00.TRP2: ACPI: PM: Power state change: D3cold -> D0
 ACPI: \_SB_.PC00.TRP1: ACPI: PM: Power state change: D3cold -> D0
-ACPI: \_SB_.PC00.TBT1: ACPI: PM: Power resource already on
+ACPI: \_SB_.PC00.I2C0: ACPI: PM: Power state change: D3hot -> D0
+ACPI: \_SB_.PC00.D3C_: ACPI: PM: Power resource already on
 ACPI: \_SB_.PC00.D3C_: ACPI: PM: Power resource already on
-ACPI: \_SB_.PC00.TBT1: ACPI: PM: Power resource already on
 ACPI: \_SB_.PC00.D3C_: ACPI: PM: Power resource already on
 ACPI: \_SB_.PC00.TBT0: ACPI: PM: Power resource already on
-acpi device:af: Power state changed to D0
-acpi device:ad: Power state changed to D0
+acpi device:84: Power state changed to D0
 acpi device:ab: Power state changed to D0
-pcieport 0000:00:07.2: Wakeup disabled by ACPI
-pcieport 0000:00:07.3: Wakeup disabled by ACPI
 pcieport 0000:00:07.1: Wakeup disabled by ACPI
 ACPI: GPE event 0x66
+ACPI: \_SB_.PC00.TBT1: ACPI: PM: Power resource turned on
+pcieport 0000:00:07.2: runtime resuming because [LNXPOWER:11] turned on
+pcieport 0000:00:07.3: runtime resuming because [LNXPOWER:11] turned on
+thunderbolt 0000:00:0d.3: runtime resuming because [LNXPOWER:11] turned on
+ACPI: \_SB_.PC00.TBT1: ACPI: PM: Power resource already on
+ACPI: \_SB_.PC00.TBT1: ACPI: PM: Power resource already on
+acpi device:a8: Power state changed to D0
+thunderbolt 0000:00:0d.3: Wakeup disabled by ACPI
+thunderbolt 0000:00:0d.3: control channel starting...
+thunderbolt 0000:00:0d.3: starting TX ring 0
+thunderbolt 0000:00:0d.3: enabling interrupt at register 0x38200 bit 0 (0x0 -> 0x1)
+thunderbolt 0000:00:0d.3: starting RX ring 0
+thunderbolt 0000:00:0d.3: enabling interrupt at register 0x38200 bit 12 (0x1 -> 0x1001)
+thunderbolt 0000:00:0d.3: ICM rtd3 veto=0x00000000
+thunderbolt 0000:00:0d.3: ICM rtd3 veto=0x00000001
+acpi device:ad: Power state changed to D0
+acpi device:af: Power state changed to D0
+pcieport 0000:00:07.3: Wakeup disabled by ACPI
+pcieport 0000:00:07.2: Wakeup disabled by ACPI
 ACPI: \_SB_.PC00.GFX0: ACPI: PM: Power state change: D0 -> D3hot
 video LNXVIDEO:00: Power state changed to D3hot
 ACPI: \_SB_.PC00.HDAS: ACPI: PM: Power state change: D0 -> D3hot
 ACPI: \_SB_.PC00.PAUD: ACPI: PM: Power resource turned off
 acpi device:35: Power state changed to D3hot
 ACPI: EC: interrupt blocked
+thunderbolt 0000:00:0d.3: stopping RX ring 0
+thunderbolt 0000:00:0d.3: disabling interrupt at register 0x38200 bit 12 (0x1001 -> 0x1)
+thunderbolt 0000:00:0d.3: stopping TX ring 0
+thunderbolt 0000:00:0d.3: disabling interrupt at register 0x38200 bit 0 (0x1 -> 0x0)
+thunderbolt 0000:00:0d.3: control channel stopped
+thunderbolt 0000:00:0d.2: stopping RX ring 0
+thunderbolt 0000:00:0d.2: disabling interrupt at register 0x38200 bit 12 (0x1001 -> 0x1)
+thunderbolt 0000:00:0d.2: stopping TX ring 0
+thunderbolt 0000:00:0d.2: disabling interrupt at register 0x38200 bit 0 (0x1 -> 0x0)
+thunderbolt 0000:00:0d.2: control channel stopped
 ACPI: PM: Preparing to enter system sleep state S4
 ACPI: EC: Stopping EC
 ACPI: EC: event blocked
@@ -67,8 +95,8 @@
 smpboot: CPU 6 is now offline
 smpboot: CPU 7 is now offline
 PM: hibernation: Creating image:
-PM: hibernation: Need to copy 1525945 pages
-PM: hibernation: Normal pages needed: 1525945 + 1024, available pages: 2612490
+PM: hibernation: Need to copy 1559993 pages
+PM: hibernation: Normal pages needed: 1559993 + 1024, available pages: 2578436
5

Looking inside the thunderbolt 4 code, it seems there are some mentions of enabling wakeup on connect or disconnect in there. However, the related debug messages do not show up, so I suspect that thunderbolt is not running with a software connection manager (where this code is), but instead uses a firmware connection manager.

It seems that the kernel prefers to use a software connection manager when ACPI indicates USB4 support is present and OS control is allowed, and otherwise tries to use a firmware connection manager, falling back to software only when the firmware manager cannot be initialized: linux/nhi.c at b2d229d4ddb17db541098b83524d901257e93845 · torvalds/linux · GitHub

Looking at this ACPI indication, it apparently requires both the OS indicating USB4 OS control support and getting it confirmed and the OS requesting and receiving control. Both of these use the _OSC ACPI method (but with different UUIDs as params).

Since I do not see debug prints for the latter, I’m assuming the former either fails or the firmware indicates no USB4 OS control support.

Looking in dmesg at startup, I do see this:

apr 18 13:22:25 dottie kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_TZ.ETMD], AE_NOT_FOUND (20210730/psargs-330)
apr 18 13:22:25 dottie kernel: 
apr 18 13:22:25 dottie kernel: No Local Variables are initialized for Method [_OSC]
apr 18 13:22:25 dottie kernel: 
apr 18 13:22:25 dottie kernel: Initialized Arguments for Method [_OSC]:  (4 arguments defined for method invocation)
apr 18 13:22:25 dottie kernel:   Arg0:   00000000fb231197 <Obj>           Buffer(16) 5D A8 3B B2 B7 C8 42 35
apr 18 13:22:25 dottie kernel:   Arg1:   0000000025d9a193 <Obj>           Integer 0000000000000001
apr 18 13:22:25 dottie kernel:   Arg2:   0000000078b55545 <Obj>           Integer 0000000000000002
apr 18 13:22:25 dottie kernel:   Arg3:   00000000bf92f763 <Obj>           Buffer(8) 00 00 00 00 05 00 00 00
apr 18 13:22:25 dottie kernel: 
apr 18 13:22:25 dottie kernel: ACPI Error: Aborting method \_SB.IETM._OSC due to previous error (AE_NOT_FOUND) (20210730/psparse-529)

Which supports the “OSC method is failing” theory.

I’m not entirely sure which method call this is, though. The Arg0 buffer shown seems incomplete (only 8 bytes rather than 16), and does not match either of the UUIDs documented here: 6. Device Configuration — ACPI Specification 6.4 documentation

Also, if this failure just means falling back to a firmware connection manager, that does not necessarily explain why the wakeup happens, it just means that that might be more likely to be a firmware/BIOS bug rather than an OS bug maybe?

6

The ACPI error turns out to be a red herring, since it is about the _SB.IETM._OSC method, which is something completely different from the _SB._OSC method that is used for negotiating USB4 support (see BIOS bug: `_TZ.ETMD` missing in ACPI tables - #2 by Matthijs_Kooijman for further discussion of that ACPI error).

However, looking at the decompiled ACPI table for the _SB._OSC method, it seems that that only (sometimes) clears bit 2 (PR3 support) and leaves all others as supported, so I think the first part of the native USB4 check should be satisfied.

However, the second part is where the kernel actually requests control over the USB tunneling using _SB._OSC with UUID 23A0D13A-26AB-486C-9C5F-0FFA525A575A (see linux/bus.c at b2d229d4ddb17db541098b83524d901257e93845 · torvalds/linux · GitHub) but it seems that the ACPI tables do not actually handle this method (so they return an unsupported UUID, which is probably silently discarded by the kernel).

So, I guess that means that the system / ACPI tables do not support using a software connection manager, so any fix must be in the “firmware” connection manager (though I’m not quite sure where this firmware actually runs - is there some kind of MCU that handles this, or is this firmware on the EC?).

7

I looked around the EC firmware a bit (which is open-source), and I’m pretty sure this “firmware connection manager” does not run in there. I suspect there is just an actual MCU or so inside the Tiger Lake CPU inside the TB controllers that handles this. Also, the kernel sources suggest that (unlike earlier xxx ridge external TB chips) tiger lake does not have updateable NVM for firmware, so maybe this firmware is hardcoded in ROM inside the CPU (or maybe flashed as part of microcode, BIOS, or some other non-TB-specific firmware?).

I did find some code inside the EC that handles the power-on-AC-attach, so the EC is at least capable of turning the system on (obviously, since it also handles the power button). However, it seems not all powering on runs through the EC, since I also noticed that when no AC power is present, the EC really powers itself down by cutting off its own power supply, rather than neatly going into deep sleep with some interrupt wake up sources left on suggested in this comment. It does use the internal VCI module to allow powering itself back on when a button press happens (using two GPIO lines from the “power button” on the mainboard, or the “fp” fingerprint-powerbutton), but this does not seem to handle power on on AC-attach - so there must be some external circuitry that handles powering on at least the EC on EC attach (and maybe also the entire system and handle the power-on-AC setting, since I could not find any evidence in the EC firmware source that it actually triggers the HOOK_AC_CHANGE on startup, so if plugging in power would only start the EC, it would never seem to power on the system - but I’m probably missing something here).

In any case, I wondered if it is also the EC that turns the system on in the problematic case. Probably not, but I had hoped to confirm this. I did find the ectool console command (with the customized-for-framework ectool version from Exploring the Embedded Controller), but the EC produces too much output during boot, so I cannot see the output from the actual power-up moment, unfortunately…

I’ve also tried looking for wakeup sources in the thunderbolt kernel module, but not much luck. I did find two possibly relevant wakeup sources in powertop: My keyboard, and the ethernet adapter. Disabling these and then hibernating prevented wakeup using the keyboard, as expected, but did not prevent wakeup when unplugging the dock, so that’s not it… (the wakeup-on-keyboard thing did make me wonder how much logic and power supplies are left running when you hibernate - it seems like the system isn’t fully off, but the CPU is still in some sleep state with the USB controller still running or sleeping, so I’m curious how much power is still used. Not much of a problem in this particular case, since there will be wall power if there is a keyboard, but still. And of course, when disconnecting wall power, such extra drain should maybe be removed, so maybe that’s why this wakeup-on-disconnect is implemented by the firmware?).

8

Just another data point, if it helps. I see similar behavior in Windows. If I hibernate the laptop, and it is still powered externally, I can wake it up with a mouse movement or a key press on an external keyboard (connected on a different USB port). If the external power is removed (while hibernated), then the external mouse/keyboard actions won’t wake up the laptop.

9

@Matthijs_Kooijman
This is interesting, i suspect when you are hibernating in this case, the system is only going to S4 which will keep the laptop partially on, and allow the cpu to wake up the laptop for various reasons.

2 Likes
10

Yup, that seems correct. Dmesg shows:

ACPI: PM: Preparing to enter system sleep state S4

That is not a problem by itself (as long as power usage is low enough and/or still plugged into wall power), and is even somewhat useful if you can still power on the system using an external keyboard (though in my workflow, powering up my dock and using the power-on-AC feature removes the need for using the keyboard).

I’ll see if I can figure out where it is decided to use S4, and if I can maybe hibernate to an even lower power state. If that works, that would solve my direct problem, but I’d still consider waking up on unplug in S4 unexpected behavior, but not sure where a fix would be…

11

You can set HibernateMode=shutdown in /etc/systemd/sleep.conf, which should trigger a full poweroff (rather than S4 as is the default). I should note that if you’re using a desktop environment like Gnome or KDE, it may override the systemd settings and you’ll have to set the relevant option(s) in the desktop environment power settings.

12

Thanks, I’ll try that. For now, I’m just using manual systemctl hibernate so that should be the right place (still have to see how to tell Gnome to support hibernate, IIRC it requires some dconf-twiddling because it’s not enabled by default).

13

I finally got around to trying this (I wanted to do some power measurements in the default configuration to compare with first).

The default for HibernateMode is platform shutdown (i.e. platform and fallback to shutdown), meaning to try a platform-specific sequence (i.e. that likely results in S4), and if that fails, try a regular shutdown (these values are written to /sys/power/disk, see https://www.kernel.org/doc/Documentation/power/interface.txt).

I have now changed this into HibernateMode=shutdown, which indeed resolves the problem. I can hibernate with my dock attached, and the laptop stays off when I unplug the USB-C cable.

As for power usage: It seems that with S4 and with shutdown, the power usage is negligable (percentage was equal or 1% lower, sometimes the battery voltage would even be higher than before the hibernate). I was afraid of running into Battery slowly drains when completely shut down when switching to shutdown mode, but maybe that issue does not occur at all on my laptop for some reason (it should also be fixed in the latest BIOS release, which I have not tried yet, so all this testing was using 3.07).

Thanks all, for helping diagnose this!

6 Likes