[TRACKING] TB3 NVMe SSD enclosure shows up if connected on boot, but not when hot-plugging

Cross-posted from Stack Exchange, because I’m beginning to think it’s more of a “Framework issue” than a “Linux issue”.

Updating the bios to 3.06 didn’t help, either.


I recently purchased an ACASIS NVMe SSD thunderbolt enclosure (Intel JHL7540 chipset).

Running on a Intel 12th gen framework laptop, the device shows up in fdisk -l and similar when the device is attached during boot, but if the device is plugged in any time after boot, it isn’t detected.

I’m running NixOS, but booted the laptop into a live-usb Ubuntu 22.10 and saw the same behavior. When plugging into a 2019 MacBook Pro, the drive shows up as expected; similarly, if I plug into a Google Pixel 5, the drive is detected as well. Thus, my conclusion is that either

  • Hotplugging a device of this type is not supported on Linux; either thunderbolt isn’t hotpluggable-in-general, or thunderbolt storage isn’t.
  • There is an issue with hotplugging on the Framework Laptop (my understanding is that Thunderbolt “closer to the metal” and thus can be effect more by vendor firmware)
  • This particular ACASIS device is doing something strange. If so, I can try a different enclosure – but I can’t find reports of whether or not other folks have been successful.

I’ve gone through many posts and articles on the topic, including:

I’ve posted what I believe to be the most relevant outputs below, but I think the best lead is one that shows up in red in the colorized dmesg output:

[ 3011.809279] pcieport 0000:03:04.0: pciehp: Hotplug bridge without secondary bus, ignoring

Any tips?


Other Logs

boltctl:

 ● ACASIS TBU405
   ├─ type:          peripheral
   ├─ name:          TBU405
   ├─ vendor:        ACASIS
   ├─ uuid:          c4010000-0072-740e-0343-90dd1021fb08
   ├─ generation:    Thunderbolt 3
   ├─ status:        authorized
   │  ├─ domain:     00278780-803d-f105-ffff-ffffffffffff
   │  ├─ rx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  ├─ tx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  └─ authflags:  none
   ├─ authorized:    Fri 17 Mar 2023 11:42:47 PM UTC
   ├─ connected:     Fri 17 Mar 2023 11:42:47 PM UTC
   └─ stored:        Fri 17 Mar 2023 10:45:35 PM UTC
      ├─ policy:     iommu
      └─ key:        no

lspci, showing on the diff from when the enclosure in plugged in:

02:00.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] (rev 06)
03:01.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] (rev 06)
03:02.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] (rev 06)
03:04.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] (rev 06)
04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983

dmesg, when device is plugged in:

[ 2918.207025] thunderbolt 0-0:1.1: new retimer found, vendor=0x8087 device=0x15ee
[ 2918.763857] thunderbolt 0-1: new device found, vendor=0x25 device=0x1
[ 2918.763863] thunderbolt 0-1: ACASIS TBU405
[ 2918.786230] pcieport 0000:00:07.0: pciehp: Slot(3): Card present
[ 2918.916774] pci 0000:02:00.0: [8086:15ef] type 01 class 0x060400
[ 2918.916850] pci 0000:02:00.0: enabling Extended Tags
[ 2918.917036] pci 0000:02:00.0: supports D1 D2
[ 2918.917037] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 2918.917153] pci 0000:02:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x4 link at 0000:00:07.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[ 2918.917406] pci 0000:02:00.0: Adding to iommu group 17
[ 2918.917527] pcieport 0000:00:07.0: ASPM: current common clock configuration is inconsistent, reconfiguring
[ 2918.919724] pci 0000:02:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 2918.919868] pci 0000:03:01.0: [8086:15ef] type 01 class 0x060400
[ 2918.919942] pci 0000:03:01.0: enabling Extended Tags
[ 2918.920105] pci 0000:03:01.0: supports D1 D2
[ 2918.920106] pci 0000:03:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 2918.920325] pci 0000:03:01.0: Adding to iommu group 18
[ 2918.920458] pci 0000:03:02.0: [8086:15ef] type 01 class 0x060400
[ 2918.920531] pci 0000:03:02.0: enabling Extended Tags
[ 2918.920694] pci 0000:03:02.0: supports D1 D2
[ 2918.920696] pci 0000:03:02.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 2918.921179] pci 0000:03:02.0: Adding to iommu group 19
[ 2918.921309] pci 0000:03:04.0: [8086:15ef] type 01 class 0x060400
[ 2918.921381] pci 0000:03:04.0: enabling Extended Tags
[ 2918.921539] pci 0000:03:04.0: supports D1 D2
[ 2918.921540] pci 0000:03:04.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 2918.921763] pci 0000:03:04.0: Adding to iommu group 20
[ 2918.921918] pci 0000:02:00.0: PCI bridge to [bus 03-2a]
[ 2918.921929] pci 0000:02:00.0:   bridge window [io  0x0000-0x0fff]
[ 2918.921934] pci 0000:02:00.0:   bridge window [mem 0x00000000-0x000fffff]
[ 2918.921943] pci 0000:02:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[ 2918.921946] pci 0000:03:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 2918.921957] pci 0000:03:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 2918.921967] pci 0000:03:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 2918.922089] pci 0000:04:00.0: [144d:a808] type 00 class 0x010802
[ 2918.922128] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[ 2918.922448] pci 0000:04:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x4 link at 0000:00:07.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[ 2918.922612] pci 0000:04:00.0: Adding to iommu group 21
[ 2918.926730] pci 0000:03:01.0: PCI bridge to [bus 04-2a]
[ 2918.926739] pci 0000:03:01.0:   bridge window [io  0x0000-0x0fff]
[ 2918.926744] pci 0000:03:01.0:   bridge window [mem 0x00000000-0x000fffff]
[ 2918.926752] pci 0000:03:01.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
[ 2918.926755] pci_bus 0000:04: busn_res: [bus 04-2a] end is updated to 2a
[ 2918.926785] pci_bus 0000:2b: busn_res: [bus 2b-53] end is updated to 53
[ 2918.926789] pci 0000:03:02.0: devices behind bridge are unusable because [bus 2b-53] cannot be assigned for them
[ 2918.926815] pci_bus 0000:54: busn_res: [bus 54-7c] end is updated to 7c
[ 2918.926818] pci 0000:03:04.0: devices behind bridge are unusable because [bus 54-7c] cannot be assigned for them
[ 2918.926821] pci_bus 0000:03: busn_res: [bus 03-2a] end can not be updated to 7c
[ 2918.926829] pci 0000:03:01.0: bridge window [mem 0x00100000-0x001fffff 64bit pref] to [bus 04-2a] add_size 100000 add_align 100000
[ 2918.926831] pci 0000:03:01.0: bridge window [mem 0x00100000-0x001fffff] to [bus 04-2a] add_size 100000 add_align 100000
[ 2918.926834] pci 0000:02:00.0: bridge window [mem 0x00100000-0x001fffff 64bit pref] to [bus 03-2a] add_size 100000 add_align 100000
[ 2918.926836] pci 0000:02:00.0: bridge window [mem 0x00100000-0x001fffff] to [bus 03-2a] add_size 100000 add_align 100000
[ 2918.926842] pci 0000:02:00.0: BAR 14: assigned [mem 0x7c000000-0x881fffff]
[ 2918.926845] pci 0000:02:00.0: BAR 15: assigned [mem 0x6000000000-0x601bffffff 64bit pref]
[ 2918.926846] pci 0000:02:00.0: BAR 13: assigned [io  0x4000-0x6fff]
[ 2918.926849] pci 0000:03:01.0: BAR 14: assigned [mem 0x7c000000-0x81ffffff]
[ 2918.926850] pci 0000:03:01.0: BAR 15: assigned [mem 0x6000000000-0x600defffff 64bit pref]
[ 2918.926852] pci 0000:03:01.0: BAR 13: assigned [io  0x4000-0x47ff]
[ 2918.926854] pci 0000:04:00.0: BAR 0: assigned [mem 0x7c000000-0x7c003fff 64bit]
[ 2918.926874] pci 0000:03:01.0: PCI bridge to [bus 04-2a]
[ 2918.926877] pci 0000:03:01.0:   bridge window [io  0x4000-0x47ff]
[ 2918.926883] pci 0000:03:01.0:   bridge window [mem 0x7c000000-0x81ffffff]
[ 2918.926888] pci 0000:03:01.0:   bridge window [mem 0x6000000000-0x600defffff 64bit pref]
[ 2918.926897] pci 0000:02:00.0: PCI bridge to [bus 03-2a]
[ 2918.926900] pci 0000:02:00.0:   bridge window [io  0x4000-0x6fff]
[ 2918.926906] pci 0000:02:00.0:   bridge window [mem 0x7c000000-0x881fffff]
[ 2918.926911] pci 0000:02:00.0:   bridge window [mem 0x6000000000-0x601bffffff 64bit pref]
[ 2918.926919] pcieport 0000:00:07.0: PCI bridge to [bus 02-2a]
[ 2918.926921] pcieport 0000:00:07.0:   bridge window [io  0x4000-0x6fff]
[ 2918.926924] pcieport 0000:00:07.0:   bridge window [mem 0x7c000000-0x881fffff]
[ 2918.926927] pcieport 0000:00:07.0:   bridge window [mem 0x6000000000-0x601bffffff 64bit pref]
[ 2918.926957] pcieport 0000:02:00.0: enabling device (0000 -> 0003)
[ 2918.927274] pcieport 0000:03:01.0: enabling device (0000 -> 0003)
[ 2918.927472] pcieport 0000:03:01.0: pciehp: Slot #1 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl+ IbPresDis- LLActRep+
[ 2918.928580] pcieport 0000:03:04.0: pciehp: Hotplug bridge without secondary bus, ignoring
[ 2918.929543] nvme nvme5: pci function 0000:04:00.0
[ 2918.929551] nvme 0000:04:00.0: enabling device (0000 -> 0002)

and dmesg when the device is unplugged

[ 2995.585232] thunderbolt 0-1: device disconnected
[ 2995.604796] pcieport 0000:00:07.0: pciehp: Slot(3): Link Down
[ 2995.604800] pcieport 0000:00:07.0: pciehp: Slot(3): Card not present
[ 2995.604812] pcieport 0000:02:00.0: Unable to change power state from D3hot to D0, device inaccessible
[ 2995.604854] pcieport 0000:03:04.0: Unable to change power state from D3hot to D0, device inaccessible
[ 2995.605091] pcieport 0000:03:04.0: Runtime PM usage count underflow!
[ 2995.605106] pcieport 0000:03:02.0: Unable to change power state from D3hot to D0, device inaccessible
[ 2995.605231] pcieport 0000:03:01.0: Unable to change power state from D3hot to D0, device inaccessible
[ 2995.605390] pcieport 0000:03:01.0: Runtime PM usage count underflow!
[ 2995.605531] pci 0000:04:00.0: Removing from iommu group 21
[ 2995.605582] pci_bus 0000:04: busn_res: [bus 04-2a] is released
[ 2995.606382] pci 0000:03:01.0: Removing from iommu group 18
[ 2995.606877] pci 0000:03:02.0: Removing from iommu group 19
[ 2995.606986] pci 0000:03:04.0: Removing from iommu group 20
[ 2995.607025] pci_bus 0000:03: busn_res: [bus 03-2a] is released
[ 2995.611720] pci 0000:02:00.0: Removing from iommu group 17

Well, that’s weird, everything looks alright to me.

So you’re saying lsblk does not list the NVME?

Is the NVME attached correctly inside that enclosure?

This is correct: lsblk behaves the same as fdisk, where the NVMe will only show up if the enclosure is attached during boot. If it is hot plugged, there is nothing.

(This, combined with trying on a macbook and android phone, is how I know the drive is attached correctly to the enclosure, and that the enclosure itself isn’t totally broken.)

I have an 11th Gen i7-1165G7 Framework with bios 3.07 running Arch Linux with an Acasis TB-401U Thunderbolt enclosure and a Samsung 980 Pro.

It works fine with hotplugging and unplugging the Acasis TB enclosure. The Framework can also boot from it.

I realise this is quite a different configuration to your 12th Gen and TB-405 enclosure but the TB-401U is also based on the Intel JHL7540 chipset.

I don’t get the messages in dmesg about:

devices behind bridge are unusable because [bus 2b-53] cannot be assigned for them

Hotplug bridge without secondary bus, ignoring

so I think that perhaps might be a good clue.

An OWC Envoy Express TB enclosure (Intel JHL6240 based) with a Samsung 970 EVO also works though it is not as nice as the Acasis.

Right, I think I’ll try to chase down that error more.

Searching for that message brings up something in the pciehp driver for the linux kernel:

	if (!dev->port->subordinate) {
		/* Can happen if we run out of bus numbers during probe */
		pci_err(dev->port,
			"Hotplug bridge without secondary bus, ignoring\n");
		return -ENODEV;
	}

and I’ve seen things other posts that seems like they may be related, but haven’t had success applying the fixes. This may be because I’m on a more recent kernel (6.2.3), and the option names may have changed?

I’ll try playing around more with the kernel options and see what happens.

Tracking this as we don’t have the enclosure to reproduce this with. Appreciate the community jumping in here.