AMD Framework and NVMe SSD Enclosure Compatibility Investigation

[Michael_Wu]

I have some perhaps enlightening info. Basically, sometimes the enclosure with the Samsung PM951 will reset when writing a very small amount (3 x 1MB). Almost always, (2 x 1MB) will not cause a reset, though I’ve seen it happen on the right bottom port. Usually, it can get up to around 100-1000 x 1MB without resetting.

Usually when a reset it happens, it’s like clockwork. For example, writing 1 x 3MB succeeds; let it idle; and then 15 seconds later, it resets. It’s almost always 15 seconds, but I’ve seen it +/- 5 from 15 seconds (might be just a recording error from me, though).

When copying larger amounts, like 1000x1MB, it may copy 275MB and then reset. This will happen only a few seconds in, and not the usual 15 seconds after. It seems like a small write that takes 1 second, and then idles, will reset in 15 seconds. But a large write may reset just a few seconds in.

After spending all day testing because there’s so many combinations and needing to wait 15 seconds between each ._. I happened to find a working combination. I think this is the biggest clue, by far: in the left bottom port, sometimes it negotiates at 5Gbps with the Sabrent EC-SNVE Realtek RTL9120B/Samsung PM951 instead of the 10Gbps it’s supposed to. For testing, I would unplug and replug from the mainboard, not the enclosure, since it seemed like it would only renegotiate speeds this way; but I’m not sure. lsusb -tv will show the devices negotiated speed on Linux. At 5Gbps, I can write a 107GB file @ ~170MB/s over 10 minutes without issue. Which means that, for some reason, it can work successfully on this mainboard. However, when it negotiates at 10Gbps (as it should), it will reset as described above.

The Plugable Realtek RTL9120/Samsung PM951 sometimes also negotiates at 5Gbps, but for some reason it still resets as described above. Maybe a firmware update could get it to work like the Sabrent Realtek RTL9120B, but it works on my other devices negotiating at the correct 10Gbps, and I don’t believe this is an enclosure issue.

The Plugable JMicron JMS583/Samsung PM951 I can’t seem to ever get to negotiate at 5Gbps on that port.

Notably, that left bottom port is one of the USB3.2 and not USB4 ports. However, I can’t seem to get any enclosure to negotiate at 5Gbps on the other ports, including the right bottom USB3.2 port.

So, so far this is what works with the Samsung PM951:

• That singular port/enclosure/5Gbps combination: Sabrent EC-SNVE Realtek RTL9120B (v1.32.87) in the left bottom port when it happens to mistakenly negotiate at 5Gbps.
• All 3 enclosures through my monitor’s 5Gbps USB hub.
• Forcing a USB2/480Mbps negotiation. Either with a USB-C USB2 cable, or by doing the “halfway-insert trick” where the USB3 USB-C cable is inserted into the mainboard where the enclosure is recognized, then insert all the way in. It will writes at those low speeds, but does not reset. Works on all ports. @brianshmrian tagging you here, hey again! You may find this interesting/helpful as it may be related.

Thanks @Michael_Schmid for the commands below as I was able to thoroughly test each power cable/enclosure port combination, using that and changing the size/count as needed and greatly simplified the process. Though, on my system, testing a singular amount with /dev/random was limited to 2GB, so I just used sample large files created with fallocate -l.

@Michael_Schmid

Drive: Crucial P3 Plus 2TB NVME CT2000P3PSSD8

dd-transfer with /dev/random

sudo dd if=/dev/random bs=1G count=10 of=transfer_test status=progress  
10737418240 bytes (11 GB, 10 GiB) copied, 30 s, 354 MB/s 
10+0 records in
10+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 30.3159 s, 354 MB/s

@Jason_Username_Taken can you try and see if your same model Sabrent enclosure (possibly on the same firmware) can also mistakenly negotiate at 5Gbps on the left bottom port and work without issue. If both of our mainboards do with our different SSDs, that may be a strong indicator. I’m not sure how to check for 5Gbps or 10Gbps in Windows, but ping me if needed and I’m sure I can figure out, if it’s possible.

I have also been chatting with Plugable support (super helpful and with them I’ve been able to further pinpoint). Here’s a statement from them that may help:

Plugable support has seen many power related issues from both host computers and from externally powered USB hubs with NVMe SSDs. They are very power demanding compared to for example SATA SSDs, or flash drives.

Again, I do strongly believe the root cause can possibly be fixed from Framework’s or AMD’s side? since:

• the Samsung PM951 works with all enclosures my 11th gen Intel mainboard (and other devices)
• the Samsung PM951 works with all enclosures through a monitor USB hub to my AMD mainboard
• the Samsung PM951 works with 1 enclosures on the left bottom port mistakenly negotiated at 5Gbps on my AMD mainboard
• the Samsung PM951 works with all enclosures forced at USB2 480Mbps speeds

I have an open support ticket already with Framework and will forward them this post and information.

My test results are below

I first began by transferring and noticing:

• 2 x 1MB files: okay
• 1 x 2MB: okay
• 3 x 1MB files: writes successfully, but then like clockwork will disconnect after 15 seconds, and then reconnect
• 1 x 3MB: writes successfully, but then like clockwork will disconnect after 15 seconds, and then reconnect
  Sometimes copying 1 file up to maybe 1GB is okay. But then copying 3 x 1MB files works but causes it to disconnect/reconnect in 15 seconds.

This is a sample journalctl log:

kernel: usb 6-1: USB disconnect, device number 50
systemd-homed[1410]: block device /sys/devices/pci0000:00/0000:00:08.3/0000:c3:00.3/usb6/6-1/6-1:1.0/host1/target1:0:0/1:0:0:0/block/sdb/sdb1 has been removed.
systemd-homed[1410]: block device /sys/devices/pci0000:00/0000:00:08.3/0000:c3:00.3/usb6/6-1/6-1:1.0/host1/target1:0:0/1:0:0:0/block/sdb has been removed.
udisksd[1419]: Cleaning up mount point /run/media/mwu/PM951 (device 8:17 no longer exists)
systemd[1]: run-media-mwu-PM951.mount: Deactivated successfully.
kernel: sd 1:0:0:0: [sdb] Synchronizing SCSI cache
nautilus[471183]: (../src/nautilus-list-base.c:1252):real_set_selection: runtime check failed: (files_to_find == NULL)
kernel: sd 1:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK

So I then tried every combination of power cable in one port and enclosure in another (nothing else plugged in). All tests run with all expansion bays empty and directly plugged in to the mainboard’s USB-C port or power cable plugged into a USB-C expansion card, except where noted. All ports also negotiated at 10Gbps, except where noted.

Also, for some reason, in the initial tests, it would reset usually at low amounts like 1x3MB / 1x6MB. It was different than the behavior I was seeing towards the end of my tests (but I had rebooted my system multiple times by then). Some of those combinations I retested, and writes would sometimes successfully go up to 1000x1MB. So assume this data may be off, as I was also developing my a testing methodology on the fly (noticeable in the data). I’ve concluded that with all combinations except the singular odd mistaken left bottom port negotiated at 5Gbps with the RTL9120B behave similarly: sometimes reset with low amounts like 3x1MB, and usually at around 1000x1MB. Let me know if I should rerun/test anything specific.

Data

Power cable in left top

Sabrent EC-SNVE Realtek RTL9120B (10Gbps) enclosure in:

• Right top: 3 x 1MB transfers but causes it to reset after 15 seconds.
• Right bottom: Writing 3 x 1MB was okay. Tried writing 3 x 100MB which caused it to reset. Then I tried writing 3 x 1MB which reset it in 15 seconds.
• Left bottom: same 15 second reset
  • I tested it the day after (10Gbps) and it would go up to 1000 x 1MB.

Power cable in left bottom

Sabrent EC-SNVE Realtek RTL9120B (10Gbps) enclosure in:

Left top

Same (not sure why I wrote this, I think I meant similar results to the Right bottom results below).

Right top:

Same (not sure why I wrote this, I think I meant similar results to the Right bottom results below).

Right bottom:

1. Writing 3 x 1MB did not result in a reset.
2. Writing 4 x 1MB did not result in a reset.
3. Writing 10 x 4MB resulted in a reset in 15 seconds. FAIL
4. Then writing 3x1MB resulted in a reset. FAIL
5. Writing 2x1MB did not result in a reset.
6. Writing 3x1MB twice, no reset.
7. Writing 4x1MB, no reset.
8. Writing 5x1MB, no reset.
9. Writing 6x1MB, reset. FAIL
10. Writing 2x1MB, no reset.
11. Writing 3x1MB, no reset.
12. Writing 4x1MB, no reset.
13. Writing 5x1MB, no reset.
14. Writing 6x1MB, reset in >20 seconds. FAIL
15. Writing 3x1MB, reset.

Power cable in right top port

Sabrent EC-SNVE Realtek RTL9120B (10Gbps) enclosure in:

Left top:

1. 3 x 1MB, no reset.
2. 4 x 1MB, no reset.
3. 5 x 1MB, no reset.
4. 6 x 1MB, reset. FAIL
5. 3 x 1MB, reset. FAIL
   I tested this particular combination the day after and it would go up to 500 x 1MB without resetting

Left bottom:

3 x 1MB, reset. FAIL

Right bottom:

3 x 1MB, reset. FAIL

Power cable in right bottom port

Sabrent EC-SNVE Realtek RTL9120B (10Gbps) enclosure in:

Left top:

1. 3 x 1MB, no reset.
2. 3 x 1MB, no reset.
3. 4 x 1MB, no reset.
4. 5 x 1MB, no reset.
5. 5 x 1MB, no reset.
6. 6 x 1MB, no reset.
7. 6 x 1MB, no reset.
8. 7 x 1MB, reset in 17 seconds.

Left bottom (I noticed it negotiated/connected at 5Gbps instead of the usual 10Gbps so I tested with that):

1. 3 x 1MB, no reset.
2. 3 x 1MB, no reset.
3. 4 x 1MB, no reset.
4. 5 x 1MB, no reset.
5. 6 x 1MB, no reset.
6. 7 x 1MB, no reset.
7. 8 x 1MB, no reset
8. 10 x 1MB, no reset
9. 20 x 1MB, no reset
10. 50 x 1MB, no reset
11. 100 x 1MB, no reset
12. 500 x 1MB, no reset
13. 1000 x 1MB, no reset
14. 5000 x 1MB, no reset
15. 10000 x 1MB, no reset
16. 20000 x 1MB, no reset (took 124.5 seconds at ~170MB/s, no reset) SUCCESS
17. 1 x 100MB, no reset
18. 1 x 500MB, no reset
19. 1 x 1,000MB, no reset
20. 1 x 5GB no reset
21. 1 x 10GB no reset
22. 1 x 107GB surprisingly no reset, it took 10 minutes 2 seconds. The entire file wrote successfully at ~190MB/s. SUCCESS. I also tested if this worked on battery power, not charging, with PCIe ASPM set to powersupersave, and a power savings profile to see if it made a difference. In theory, that should be the least reliable. Note I did not have such power savings settings in use for other tests. Writes were slower at ~60MB/s for the first 10% for a minute or two (where it usually resets a few seconds in) and it didn’t reset, so I assumed it worked as well. I changed back to power profile/settings I have while running on AC, but without the charger. Speed went up to ~150MB/s and finished successfully.
23. I then disconnected the enclosure and reconnected it into the same port with the same cable, and it negotiated at 10Gbps (as it should):
24. 3 x 1MB: no reset
25. 3 x 1MB: no reset
26. 4 x 1MB: no reset
27. 5 x 1MB: no reset
28. 6 x 1MB: no reset
29. 7 x 1MB: no reset
30. 8 x 1MB: no reset
31. 10 x 1MB: no reset
32. 20 x 1MB: no reset
33. 50 x 1MB: no reset
34. 100 x 1MB: no reset
35. 500 x 1MB: no reset
36. 1000 x 1MB: no reset
37. 5000 x 1MB: reset. FAIL
38. 1000 x 1MB: reset. FAIL
39. 3 x 1MB: no reset
40. 100 x 1MB: no reset
41. 500 x 1MB: no reset
42. 1000 x 1MB: no reset
43. 5000 x 1MB: reset. FAIL
44. 3 x 1MB: no reset
45. 100 x 1MB: no reset
46. 500 x 1MB: no reset
47. 1000 x 1MB: reset. FAIL
48. 3 x 1MB: no reset
49. 100 x 1MB: no reset
50. 500 x 1MB: no reset
51. 1000 x 1MB: reset. FAIL
52. Disconnected and reconnected enclosure cable physically and got it to negotiate at 5Gbps again to test:
53. 3 x 1MB: no reset
54. 3 x 1MB: no reset
55. 4 x 1MB: no reset
56. 5 x 1MB: no reset
57. 6 x 1MB: no reset
58. 7 x 1MB: no reset
59. 8 x 1MB: no reset
60. 10 x 1MB: no reset
61. 20 x 1MB: no reset
62. 50 x 1MB: no reset
63. 100 x 1MB: no reset
64. 500 x 1MB: no reset
65. 1000 x 1MB: no reset
66. 5000 x 5000MB: no reset
67. 10000 x 1MB: no reset
68. 1 x 100MB: no reset
69. 1 x 500MB: no reset
70. 1 x 1,000MB: no reset
71. 1 x 5GB: no reset
72. 1 x 10GB: no reset
73. 1 x 107GB no reset. Took 10 minutes 10 seconds, ~168MB/S. SUCCESS

Right top:

1. 3 x 1MB: no reset
2. 10 x 1MB: reset. FAIL
3. 3 x 1MB: reset. FAIL

No power cable, on battery

I accidentally left a USB-C expansion card in the right top slot but I don’t think it affected anything.

Sabrent EC-SNVE Realtek RTL9120B (10Gbps) enclosure in:

Left top port:

1. 2 x 1MB: no reset
2. 3 x 1MB: no reset
3. 5 x 1MB: no reset
4. 10 x 1MB: no reset
5. 20 x 1MB: no reset
6. 100 x 1MB: no reset
7. 500 x 1MB: no reset
8. 1000 x 1MB: no reset
9. 5000 x 1MB: no reset
10. 10000 x 1MB: reset 7.2GB in
11. 3 x 1MB: no reset
12. 10 x 1MB: no reset
13. 50 x 1MB: no reset
14. 100 x 1MB no reset
15. 500 x 1MB: no reset
16. 1000 x 1MB: no reset
17. 5000 x 1MB: reset 3.3GB in

Left bottom port

1. 2 x 1MB: no reset
2. 3 x 1MB: no reset
3. 5 x 1MB: no reset
4. 10 x 1MB: no reset
5. 20 x 1MB: no reset
6. 100 x 1MB: no reset
7. 500 x 1MB: no reset
8. 1000 x 1MB: no reset
9. 5000 x 1MB: reset in 3.3GB
10. 3 x 1MB:
11. 10 x 1MB: no reset
12. 50 x 1MB: no reset
13. 100 x 1MB no reset
14. 500 x 1MB: reset

This port does work with Sabrent RTL9120B/Samsung PM951 combination mistakenly negotiated at 5Gbps on battery power, as described in the batch of tests above this.

Right top port:

1. 2 x 1MB: no reset
2. 3 x 1MB: no reset
3. 5 x 1MB: no reset
4. 10 x 1MB: no reset
5. 20 x 1MB: no reset
6. 100 x 1MB: no reset
7. 500 x 1MB: no reset
8. 1000 x 1MB: no reset
9. 5000 x 1MB: reset at 3.2GB
10. 3 x 1MB: no reset
11. 10 x 1MB: no reset
12. 50 x 1MB: no reset
13. 100 x 1MB no reset
14. 500 x 1MB: reset

Right bottom port:

1. 2 x 1MB: no reset
2. 3 x 1MB: no reset
3. 5 x 1MB: no reset
4. 10 x 1MB: no reset
5. 20 x 1MB: no reset
6. 100 x 1MB: no reset
7. 500 x 1MB: no reset
8. 1000 x 1MB: no reset
9. 5000 x 1MB: reset 2.6GB in
10. 3 x 1MB: no reset
11. 10 x 1MB: no reset
12. 50 x 1MB: no reset
13. 100 x 1MB: no reset
14. 500 x 1MB: no reset
15. 1000 x 1MB: no reset
16. 5000 x 1MB: reset 2.8 GB in

No power cable, on battery

Plugable RTL9120 (10Gbps) enclosure in:

Left top port:

1. 2 x 1MB: no reset
2. 3 x 1MB: no reset
3. 5 x 1MB: no reset
4. 10 x 1MB: no reset
5. 20 x 1MB: no reset
6. 100 x 1MB: no reset
7. 500 x 1MB: reset
8. 3 x 1MB: reset
9. 2 x 1MB: no reset
10. 3 x 1MB: no reset
11. 5 x 1MB: no reset
12. 50 x 1MB: no reset
13. 100 x 1MB: no reset
14. 500 x 1MB: reset

Left bottom port (negotiated at 5Gbps):

1. 2 x 1MB: no reset
2. 3 x 1MB: no reset
3. 5 x 1MB: no reset
4. 10 x 1MB: no reset
5. 20 x 1MB: no reset
6. 100 x 1MB: no reset
7. 500 x 1MB: reset
8. 3 x 1MB: reset
9. 5 x 1MB: reset
10. 50 x 1MB: no reset
11. 100 x 1MB: no reset
12. 500 x 1MB: reset

Left bottom port (negotiated at 10Gbps):

1. 2 x 1MB: no reset
2. 3 x 1MB: no reset
3. 5 x 1MB: no reset
4. 10 x 1MB: no reset
5. 20 x 1MB: no reset
6. 100 x 1MB: no reset
7. 500 x 1MB: no reset
8. 1000 x 1MB: reset
9. 3 x 1MB: reset
10. 2 x 1MB: no reset
11. 3 x 1MB: no reset
12. 5 x 1MB: no reset
13. 50 x 1MB: no reset
14. 100 x 1MB: reset

Right top port:

1. 2 x 1MB: no reset
2. 3 x 1MB: no reset
3. 5 x 1MB: no reset
4. 10 x 1MB: no reset
5. 20 x 1MB: no reset
6. 100 x 1MB: no reset
7. 500 x 1MB: reset
8. 3 x 1MB: reset
9. 2 x 1MB: no reset
10. 3 x 1MB: no reset
11. 5 x 1MB: no reset
12. 10 x 1MB: no reset
13. 50 x 1MB: no reset
14. 100 x 1MB no reset
15. 500 x 1MB: reset

Right bottom port:

1. 2 x 1MB: reset
2. 2 x 1MB: no reset
3. 3 x 1MB: no reset
4. 5 x 1MB: no reset
5. 10 x 1MB: no reset
6. 20 x 1MB: reset
7. 2 x 1MB: reset
8. 2 x 1MB: reset
9. 2 x 1MB: no reset
10. 3 x 1MB: reset
11. Physically unplugged and replugged
12. 2 x 1MB: no reset
13. 3 x 1MB: no reset
14. 10 x 1MB: no reset
15. 50 x 1MB: reset