Power supply probably isn’t an issue per se. I’m now doing some sustained transfer test with external HDD, with this particular one being an HDD+Tiny SSD Hybrid drive. I assume this probably draws more power than my Gen3 NVMe in an enclosure. (although I do not have a device to measure)
Approaching 80GBs now, no issues.
Another update. Was just messing with the laptop before ending the night, plugged external drive to Port 2, experienced ‘cool down’ and can’t recognize drive as usual, I kept the drive connected and triggered ‘battery disconnect’ in BIOS, afterward I also held down power button for ~30seconds for good measure.
After waking the computer by attaching DC to Port 1, I noticed it seems like the external enclosure isn’t getting power at all. Disconnected external enclosure and connected android phone to Port 2 instead, no data and phone shows it’s charging the laptop. Therefore, it does seems like there’s some funky negotiation happening.
Maybe related to the post below?
(Recovered to status quo by disconnecting everything, and trigger ‘battery disconnect’ again)
You could also just use an ssd that is not some form of 970 evo XD.
Also dicounting power issues, anything but a 970 evo (on linux) works in my pretty power hungry asmedia usb4 enclosure, including a PM9A1 which is a lot more power hungry than the 970 evo.
Just did a full device read (512GB) at 2.8GB/s (the bandwidth cap is still there but somehow higher than last time I tested).
Was the 970 evo plus realtec enclosure issue ever reproduced on windows?
I use a GM7000 4 TB that’s made of an entirely different controller and entirely different NAND. It does not work, nonetheless.
TIL acer makes ssds.
What kind of enclosure are you running? Also is it not working on linux or windows or both?
The issue of the NVMe enclosure dropping out is very similar to what I’m running into with my iPhone 15 Pro Max that I posted in this thread: USB C is not compatible with iPhone 15 / Pro / Pro Max - #42 by Tyrael
TLDR: C-to-C cables constantly connect/disconnect; USB 3 A-to-C cable connects fine, but drops data link after a few minutes trying to backup phone; USB 2.0 A-to-C cable works fine.
I also have a TeamGroup USB 3.2 Gen 1 flash drive that is not seen at all by Windows when I plug it in, but works on other machines fine.
It seems more like the USB controllers between the AMD Framework board and what’s on the device aren’t playing nice.
Hi ! I came across this thread and thought I could share my experience.
I use Windows To Go on a regular basis for gaming with the SATA/USB adapter and SSD. I find it convenient to separate my work space and my gaming space.
I tried Installing Windows to Go on the EL-2230C (Windows 11 Pro) with a software called Rufus. I applied the image either from the Framework Laptop itself (from a Virtualbox VM) or from a Windows To Go (Windows 11 Pro) install booted from a SATA/USB Drive.
Depending on the case, I always end up with a corrupted image on the EL-2230C:
This behavior does not occur at all with a SATA to USB adapter. I tried cloning my daily (known good) Windows To Go image from my SATA drive to the EL-2230C, but ended up in case 4 in only a few minutes.
Yup, there are other folks having issues with one-cable Type-C portable monitors not working on AMD FW, despite the same monitor works fine on Intel FW, and other computers. Defintely seems like some controllers isn’t functioning as they should.
Yup, I use Rufus created WinToGo quite often. Even if an intact WinToGo is created on the external NVMe with another computer, and successfully booting the AMD FW with it, once the R/W gets intensive it will BSOD. Essentially the drive will become unstable or drops outright when activitiy becomes intensive, irrelevant to what the activity is.
Likely not the issue especially if the drive works fine on other computers. My enclosure can become unsafe to touch at the extreme, yet sustained file transfer, WinToGo creation, and WinToGo booting all consistently work fine with my Intel Framework.
BTW. I think this is not USB-C’s problem. I use same enclosure and SSD with type-c to USB-A cable. I connect to framework official USB-A expansion card. The situation is the same.
hmm, not really. Same SSD, recently appears bad, speed 11MB/s, lots of error message…
insert into an RTL9210 (ugreen CM559) enclosure. it immediately goes
361681125376 bytes (362 GB, 337 GiB) copied, 374 s, 967 MB/s
362630086656 bytes (363 GB, 338 GiB) copied, 375 s, 967 MB/s
363569610752 bytes (364 GB, 339 GiB) copied, 376 s, 967 MB/s
364576243712 bytes (365 GB, 340 GiB) copied, 377 s, 967 MB/s
Are there any better firmware for ASM2364/2464?
The firmwares didn’t make any differences in my case.
I also didn’t have performance issues with the 970 evo it just didn’t hotplug on linux, when I pulged it in before booting it performaned just fine…
hmm with AS2464, plug before boot, the performance is good ~2GB/s. boltctl shows 20Gbps x 2 link.
But somehow if something happens to 970evo with AS2364, seemingly “triggers” some condition. using RTL9210 for some time, will somehow “heal” it.
No idea, maybe because of I added some capacitors to power of AS2364 enclosure?
Anyway, I can’t get x 2 link neither USB nor TB for devices I can reach.
With new 6.7.12 kernel and 3.05 BIOS, everything’s almost the same.
I have had this problem on a Framework 16 with a Sabrent NVMe SSD Enclosure on Alpine Linux with a Samsung SSD inside of it. It will work for 10 minutes then stop.
Oh, so it does appear that this issue is applicable to both the AMD 13 and 16.
My previous correspondence with FW regarding this issue was a month ago, at the time they are still internally escalating it. I wrote to support again yesterday to check on the progress, I’ll update the post when (and if) I have more information.
Just curious. Did you get anything from framework team?
Nope, I have not. Neither me nor other community member(s) who’s already in engineering escalation that also reached out to Framework regarding this specific issue in the past few weeks received response.
Hi,
I have 2 Sabrent nvme enclosures.
Both have RTL9210B.
One is mostly OK, but the other often disconnects.
Eventually I found out the problem one has a faulty usb connection. So it would intermittently disconnect, be it due to movement (easily reproducable) or heat changes.
So, people might just have faulty enclosure units.
My ok enclosure only works in slots 1, 2, 4. 5. It fails to appear at all in slot 3, 6. FW16 laptop.
Me and some other testing volunteers have ruled this possibility out on their enclosure.
For my enclosure+SSD+cable combination specifically, it works perfectly fine on 12th Gen Intel Framework, MacBooks (running Windows and macOS), and multiple other non-Framework computers including an 8840U-based one. It only exhibits issue on AMD FW13, which is consistent with reports from a few other users (alongside issues like Type-C display compatibility, USB-C iPhone compatibility).
Therefore a faulty enclosure doesn’t appear to be the cause, at least for me and a few other users in this post 
I got another SSD and it also has the same problem when connect to FW13 (AMD version).
SSD: Kingston KC2500
SSD enclosure chip: Realtek RTL9210
OS: NixOS 24.05
I use this to rsync data. After transmit about 100GB then it failed. Here is the dmesg log
[ 957.904982] sd 0:0:0:0: [sda] tag#29 uas_eh_abort_handler 0 uas-tag 17 inflight: CMD OUT
[ 957.904997] sd 0:0:0:0: [sda] tag#29 CDB: Write(10) 2a 00 83 50 62 00 00 00 10 00
[ 957.905139] sd 0:0:0:0: [sda] tag#28 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD
[ 957.905151] sd 0:0:0:0: [sda] tag#28 CDB: Write(10) 2a 00 83 50 60 20 00 00 08 00
[ 957.905161] sd 0:0:0:0: [sda] tag#20 uas_eh_abort_handler 0 uas-tag 23 inflight: CMD OUT
[ 957.905167] sd 0:0:0:0: [sda] tag#20 CDB: Write(10) 2a 00 83 50 63 e0 00 00 10 00
[ 957.905302] sd 0:0:0:0: [sda] tag#19 uas_eh_abort_handler 0 uas-tag 21 inflight: CMD OUT
[ 957.905311] sd 0:0:0:0: [sda] tag#19 CDB: Write(10) 2a 00 83 50 63 a0 00 00 10 00
[ 957.905422] sd 0:0:0:0: [sda] tag#18 uas_eh_abort_handler 0 uas-tag 20 inflight: CMD OUT
[ 957.905433] sd 0:0:0:0: [sda] tag#18 CDB: Write(10) 2a 00 83 50 63 38 00 00 08 00
[ 957.905540] sd 0:0:0:0: [sda] tag#17 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
[ 957.905547] sd 0:0:0:0: [sda] tag#17 CDB: Read(10) 28 00 13 0e 8f e0 00 00 70 00
[ 957.905554] sd 0:0:0:0: [sda] tag#16 uas_eh_abort_handler 0 uas-tag 22 inflight: CMD OUT
[ 957.905558] sd 0:0:0:0: [sda] tag#16 CDB: Write(10) 2a 00 83 50 63 d0 00 00 08 00
[ 957.905681] sd 0:0:0:0: [sda] tag#15 uas_eh_abort_handler 0 uas-tag 19 inflight: CMD OUT
[ 957.905692] sd 0:0:0:0: [sda] tag#15 CDB: Write(10) 2a 00 83 50 62 a0 00 00 08 00
[ 957.905794] sd 0:0:0:0: [sda] tag#14 uas_eh_abort_handler 0 uas-tag 18 inflight: CMD OUT
[ 957.905799] sd 0:0:0:0: [sda] tag#14 CDB: Write(10) 2a 00 83 50 62 78 00 00 10 00
[ 957.905888] sd 0:0:0:0: [sda] tag#13 uas_eh_abort_handler 0 uas-tag 16 inflight: CMD OUT
[ 957.905892] sd 0:0:0:0: [sda] tag#13 CDB: Write(10) 2a 00 83 50 61 f0 00 00 08 00
[ 957.905988] sd 0:0:0:0: [sda] tag#12 uas_eh_abort_handler 0 uas-tag 15 inflight: CMD OUT
[ 957.905993] sd 0:0:0:0: [sda] tag#12 CDB: Write(10) 2a 00 83 50 61 a0 00 00 08 00
[ 957.906078] sd 0:0:0:0: [sda] tag#11 uas_eh_abort_handler 0 uas-tag 14 inflight: CMD OUT
[ 957.906083] sd 0:0:0:0: [sda] tag#11 CDB: Write(10) 2a 00 83 50 61 80 00 00 18 00
[ 957.906167] sd 0:0:0:0: [sda] tag#10 uas_eh_abort_handler 0 uas-tag 13 inflight: CMD OUT
[ 957.906171] sd 0:0:0:0: [sda] tag#10 CDB: Write(10) 2a 00 83 50 61 70 00 00 08 00
[ 957.906252] sd 0:0:0:0: [sda] tag#9 uas_eh_abort_handler 0 uas-tag 12 inflight: CMD OUT
[ 957.906257] sd 0:0:0:0: [sda] tag#9 CDB: Write(10) 2a 00 83 50 61 48 00 00 08 00
[ 957.906342] sd 0:0:0:0: [sda] tag#8 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT
[ 957.906345] sd 0:0:0:0: [sda] tag#8 CDB: Write(10) 2a 00 83 50 61 20 00 00 18 00
[ 957.906418] sd 0:0:0:0: [sda] tag#7 uas_eh_abort_handler 0 uas-tag 11 inflight: CMD OUT
[ 957.906420] sd 0:0:0:0: [sda] tag#7 CDB: Write(10) 2a 00 83 50 60 f0 00 00 08 00
[ 957.906491] sd 0:0:0:0: [sda] tag#6 uas_eh_abort_handler 0 uas-tag 10 inflight: CMD OUT
[ 957.906494] sd 0:0:0:0: [sda] tag#6 CDB: Write(10) 2a 00 83 50 60 d8 00 00 10 00
[ 957.906564] sd 0:0:0:0: [sda] tag#5 uas_eh_abort_handler 0 uas-tag 9 inflight: CMD OUT
[ 957.906566] sd 0:0:0:0: [sda] tag#5 CDB: Write(10) 2a 00 83 50 60 c8 00 00 08 00
[ 957.906636] sd 0:0:0:0: [sda] tag#4 uas_eh_abort_handler 0 uas-tag 8 inflight: CMD OUT
[ 957.906638] sd 0:0:0:0: [sda] tag#4 CDB: Write(10) 2a 00 83 50 60 b8 00 00 08 00
[ 957.906708] sd 0:0:0:0: [sda] tag#3 uas_eh_abort_handler 0 uas-tag 7 inflight: CMD OUT
[ 957.906710] sd 0:0:0:0: [sda] tag#3 CDB: Write(10) 2a 00 83 50 60 a0 00 00 08 00
[ 957.906779] sd 0:0:0:0: [sda] tag#2 uas_eh_abort_handler 0 uas-tag 6 inflight: CMD OUT
[ 957.906781] sd 0:0:0:0: [sda] tag#2 CDB: Write(10) 2a 00 83 50 60 88 00 00 10 00
[ 957.906846] sd 0:0:0:0: [sda] tag#1 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD OUT
[ 957.906848] sd 0:0:0:0: [sda] tag#1 CDB: Write(10) 2a 00 83 50 60 50 00 00 10 00
[ 957.906913] sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT
[ 957.906915] sd 0:0:0:0: [sda] tag#0 CDB: Write(10) 2a 00 83 50 60 30 00 00 08 00
[ 959.761004] sd 0:0:0:0: [sda] tag#27 uas_eh_abort_handler 0 uas-tag 30 inflight: CMD OUT
[ 959.761026] sd 0:0:0:0: [sda] tag#27 CDB: Write(10) 2a 00 74 58 32 88 00 04 00 00
[ 959.761233] sd 0:0:0:0: [sda] tag#26 uas_eh_abort_handler 0 uas-tag 29 inflight: CMD OUT
[ 959.761243] sd 0:0:0:0: [sda] tag#26 CDB: Write(10) 2a 00 74 58 2e 88 00 04 00 00
[ 959.761406] sd 0:0:0:0: [sda] tag#25 uas_eh_abort_handler 0 uas-tag 28 inflight: CMD OUT
[ 959.761415] sd 0:0:0:0: [sda] tag#25 CDB: Write(10) 2a 00 74 58 2a 88 00 04 00 00
[ 959.761610] sd 0:0:0:0: [sda] tag#24 uas_eh_abort_handler 0 uas-tag 27 inflight: CMD OUT
[ 959.761619] sd 0:0:0:0: [sda] tag#24 CDB: Write(10) 2a 00 74 58 26 88 00 04 00 00
[ 959.761777] sd 0:0:0:0: [sda] tag#23 uas_eh_abort_handler 0 uas-tag 26 inflight: CMD OUT
[ 959.761789] sd 0:0:0:0: [sda] tag#23 CDB: Write(10) 2a 00 74 58 22 88 00 04 00 00
[ 959.761904] sd 0:0:0:0: [sda] tag#22 uas_eh_abort_handler 0 uas-tag 25 inflight: CMD OUT
[ 959.761910] sd 0:0:0:0: [sda] tag#22 CDB: Write(10) 2a 00 74 58 1e 88 00 04 00 00
[ 959.762064] sd 0:0:0:0: [sda] tag#21 uas_eh_abort_handler 0 uas-tag 24 inflight: CMD OUT
[ 959.762073] sd 0:0:0:0: [sda] tag#21 CDB: Write(10) 2a 00 74 58 1a 88 00 04 00 00
[ 959.770071] scsi host0: uas_eh_device_reset_handler start
[ 959.889345] usb 8-1: reset SuperSpeed Plus Gen 2x1 USB device number 2 using xhci_hcd
[ 962.942519] scsi host0: uas_eh_device_reset_handler success
[ 973.247196] sd 0:0:0:0: [sda] tag#29 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
[ 973.247209] sd 0:0:0:0: [sda] tag#29 CDB: Test Unit Ready 00 00 00 00 00 00
[ 973.247214] scsi host0: uas_eh_device_reset_handler start
[ 973.362021] usb 8-1: reset SuperSpeed Plus Gen 2x1 USB device number 2 using xhci_hcd
[ 973.441038] scsi host0: uas_eh_device_reset_handler success
[ 973.441057] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441063] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441066] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441069] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441072] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441075] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441078] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441082] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441085] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441088] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441091] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441094] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441097] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441100] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441103] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441106] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441109] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441112] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441115] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441118] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441121] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441124] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441127] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441130] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441133] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441136] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441139] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441142] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441146] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441149] sd 0:0:0:0: Device offlined - not ready after error recovery
[ 973.441231] sd 0:0:0:0: [sda] tag#21 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=43s
[ 973.441241] sd 0:0:0:0: [sda] tag#21 CDB: Write(10) 2a 00 74 58 1a 88 00 04 00 00
[ 973.441245] I/O error, dev sda, sector 1951931016 op 0x1:(WRITE) flags 0xd800 phys_seg 128 prio class 2
[ 973.441434] sd 0:0:0:0: [sda] tag#22 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=43s
[ 973.441439] sd 0:0:0:0: [sda] tag#22 CDB: Write(10) 2a 00 74 58 1e 88 00 04 00 00
[ 973.441442] I/O error, dev sda, sector 1951932040 op 0x1:(WRITE) flags 0xd800 phys_seg 128 prio class 2
[ 973.441590] sd 0:0:0:0: [sda] tag#23 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=43s
[ 973.441595] sd 0:0:0:0: [sda] tag#23 CDB: Write(10) 2a 00 74 58 22 88 00 04 00 00
[ 973.441597] I/O error, dev sda, sector 1951933064 op 0x1:(WRITE) flags 0xd800 phys_seg 128 prio class 2
[ 973.441794] sd 0:0:0:0: [sda] tag#24 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=43s
[ 973.441805] sd 0:0:0:0: [sda] tag#24 CDB: Write(10) 2a 00 74 58 26 88 00 04 00 00
[ 973.441808] I/O error, dev sda, sector 1951934088 op 0x1:(WRITE) flags 0xd800 phys_seg 128 prio class 2
[ 973.441973] sd 0:0:0:0: [sda] tag#25 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=43s
[ 973.441978] sd 0:0:0:0: [sda] tag#25 CDB: Write(10) 2a 00 74 58 2a 88 00 04 00 00
[ 973.441980] I/O error, dev sda, sector 1951935112 op 0x1:(WRITE) flags 0xd800 phys_seg 128 prio class 2
[ 973.442147] sd 0:0:0:0: [sda] tag#26 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=43s
[ 973.442151] sd 0:0:0:0: [sda] tag#26 CDB: Write(10) 2a 00 74 58 2e 88 00 04 00 00
[ 973.442154] I/O error, dev sda, sector 1951936136 op 0x1:(WRITE) flags 0xd800 phys_seg 128 prio class 2
[ 973.442364] sd 0:0:0:0: [sda] tag#27 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=43s
[ 973.442368] sd 0:0:0:0: [sda] tag#27 CDB: Write(10) 2a 00 74 58 32 88 00 04 00 00
[ 973.442371] I/O error, dev sda, sector 1951937160 op 0x1:(WRITE) flags 0xd800 phys_seg 128 prio class 2
[ 973.442543] sd 0:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=45s
[ 973.442548] sd 0:0:0:0: [sda] tag#0 CDB: Write(10) 2a 00 83 50 60 30 00 00 08 00
[ 973.442551] I/O error, dev sda, sector 2203082800 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[ 973.442559] Buffer I/O error on dev dm-0, logical block 275253510, lost async page write
[ 973.442589] sd 0:0:0:0: [sda] tag#1 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=45s
[ 973.442593] sd 0:0:0:0: [sda] tag#1 CDB: Write(10) 2a 00 83 50 60 50 00 00 10 00
[ 973.442596] I/O error, dev sda, sector 2203082832 op 0x1:(WRITE) flags 0x103000 phys_seg 2 prio class 0
[ 973.442602] Buffer I/O error on dev dm-0, logical block 275253514, lost async page write
[ 973.442609] Buffer I/O error on dev dm-0, logical block 275253515, lost async page write
[ 973.442624] sd 0:0:0:0: [sda] tag#2 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=45s
[ 973.442629] sd 0:0:0:0: [sda] tag#2 CDB: Write(10) 2a 00 83 50 60 88 00 00 10 00
[ 973.442631] I/O error, dev sda, sector 2203082888 op 0x1:(WRITE) flags 0x103000 phys_seg 2 prio class 0
[ 973.442638] Buffer I/O error on dev dm-0, logical block 275253521, lost async page write
[ 973.442646] Buffer I/O error on dev dm-0, logical block 275253522, lost async page write
[ 973.442680] Buffer I/O error on dev dm-0, logical block 275253524, lost async page write
[ 973.442692] Buffer I/O error on dev dm-0, logical block 275253527, lost async page write
[ 973.442704] Buffer I/O error on dev dm-0, logical block 275253529, lost async page write
[ 973.442716] Buffer I/O error on dev dm-0, logical block 275253531, lost async page write
[ 973.442722] Buffer I/O error on dev dm-0, logical block 275253532, lost async page write
[ 973.442935] sd 0:0:0:0: rejecting I/O to offline device
[ 973.443030] Aborting journal on device dm-0-8.
[ 973.443078] JBD2: I/O error when updating journal superblock for dm-0-8.
[ 973.443087] EXT4-fs error (device dm-0) in ext4_reserve_inode_write:5735: IO failure
[ 973.443106] EXT4-fs error (device dm-0): ext4_dirty_inode:5939: inode #68856363: comm rsync: mark_inode_dirty error
[ 973.443116] EXT4-fs error (device dm-0) in ext4_dirty_inode:5940: IO failure
[ 973.443552] EXT4-fs error (device dm-0): ext4_journal_check_start:84: comm rsync: Detected aborted journal
[ 973.443613] EXT4-fs (dm-0): I/O error while writing superblock
[ 973.443801] EXT4-fs (dm-0): I/O error while writing superblock
[ 973.443813] EXT4-fs (dm-0): Remounting filesystem read-only
[ 973.454892] EXT4-fs warning (device dm-0): dx_probe:822: inode #68829534: lblock 0: comm rsync: error -5 reading directory block
[ 973.515420] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #68830768: lblock 0: comm rsync: error -5 reading directory block
[ 973.759360] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #68832314: lblock 0: comm rsync: error -5 reading directory block
[ 973.804461] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #68832750: lblock 0: comm rsync: error -5 reading directory block
[ 973.810746] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #68832885: lblock 0: comm rsync: error -5 reading directory block
[ 973.830473] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #68833159: lblock 0: comm rsync: error -5 reading directory block
[ 974.028463] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #68835909: lblock 0: comm rsync: error -5 reading directory block
[ 974.051661] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #68836587: lblock 0: comm rsync: error -5 reading directory block
[ 974.051739] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #68836610: lblock 0: comm rsync: error -5 reading directory block
[ 974.228818] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #68838694: lblock 0: comm rsync: error -5 reading directory block
[ 978.463600] blk_print_req_error: 1422 callbacks suppressed
[ 978.463608] I/O error, dev sda, sector 2157027688 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 978.463633] EXT4-fs warning: 1212 callbacks suppressed
[ 978.463636] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67389580: lblock 0: comm rsync: error -5 reading directory block
[ 978.484645] I/O error, dev sda, sector 2157027792 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 978.484683] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67390054: lblock 0: comm rsync: error -5 reading directory block
[ 978.485389] I/O error, dev sda, sector 2157027816 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 978.485405] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67390074: lblock 0: comm rsync: error -5 reading directory block
[ 978.490835] I/O error, dev sda, sector 2157027864 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 978.490858] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67390227: lblock 0: comm rsync: error -5 reading directory block
[ 978.497017] I/O error, dev sda, sector 2157028096 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 978.497034] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67390631: lblock 0: comm rsync: error -5 reading directory block
[ 978.497274] I/O error, dev sda, sector 2156996752 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 978.497288] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67381735: lblock 0: comm rsync: error -5 reading directory block
[ 978.497644] I/O error, dev sda, sector 2156996696 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 978.497658] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67381713: lblock 0: comm rsync: error -5 reading directory block
[ 978.497705] I/O error, dev sda, sector 2156996672 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 978.497717] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67381710: lblock 0: comm rsync: error -5 reading directory block
[ 978.497869] I/O error, dev sda, sector 2156996712 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 978.497882] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67381720: lblock 0: comm rsync: error -5 reading directory block
[ 978.498213] I/O error, dev sda, sector 2156996480 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 978.498226] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67381088: lblock 0: comm rsync: error -5 reading directory block
[ 983.468799] blk_print_req_error: 2152 callbacks suppressed
[ 983.468803] I/O error, dev sda, sector 2157004504 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 983.468813] EXT4-fs warning: 2018 callbacks suppressed
[ 983.468814] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67392622: lblock 0: comm rsync: error -5 reading directory block
[ 983.468912] I/O error, dev sda, sector 2157004520 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 983.468917] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67392629: lblock 0: comm rsync: error -5 reading directory block
[ 983.469420] I/O error, dev sda, sector 2157004600 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 983.469426] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67392679: lblock 0: comm rsync: error -5 reading directory block
[ 983.469768] I/O error, dev sda, sector 2157004696 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 983.469772] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67392728: lblock 0: comm rsync: error -5 reading directory block
[ 983.469787] I/O error, dev sda, sector 2157004704 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 983.469791] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67392731: lblock 0: comm rsync: error -5 reading directory block
[ 983.470117] I/O error, dev sda, sector 2157004728 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 983.470121] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67392736: lblock 0: comm rsync: error -5 reading directory block
[ 983.470135] I/O error, dev sda, sector 2157004736 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 983.470139] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67392739: lblock 0: comm rsync: error -5 reading directory block
[ 983.470204] I/O error, dev sda, sector 2157004752 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 983.470207] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67392747: lblock 0: comm rsync: error -5 reading directory block
[ 983.470842] I/O error, dev sda, sector 2157004840 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 983.470846] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67392823: lblock 0: comm rsync: error -5 reading directory block
[ 983.470983] I/O error, dev sda, sector 2157004864 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 983.470987] EXT4-fs warning (device dm-0): htree_dirblock_to_tree:1082: inode #67392835: lblock 0: comm rsync: error -5 reading directory block
I think it would help to know which expansion card slot the NVME enclosure was plugged into at the time. I think some slots are better than others.