[RESPONDED] 1TB expansion card disconnects randomly

Are we feeling like 6.1 and 6.2.0-rc2 have resolved the disconnect issue? Been spending this morning tracking down Fedora bugs, I’d like to call this one solved. Please confirm - thanks. :slight_smile:

Right now, I can give you absence of evidence: I have not seen a disconnect that broke my system since switching to 6.1 stable.

Of course, that means I can’t prove evidence of absence, that it’s absolutely fixed! Nobody else seems to have chimed in, so I’d love to see more evidence from people who have been plagued by the issue on Linux!

I installed 6.1 as soon as I saw this thread, rebooted into it, and my SD card disconnected again within half an hour. :frowning:

My system is an 11th gen mainboard, BIOS 3.17 (the one that just hit LVFS), KUbuntu 22.04, kernel 6.1.4:

$ uname -r
6.1.4-060104-generic

The BIOS settings I have that might be relevant: the battery is capped at 80%, and I have this:

bios -> Advanced -> Boot performance mode = Max Non-Turbo Performance

thanks to this comment: 1TB expansion card disconnects randomly - #12 by Shy_Guy . Setting that a few months ago reduced the disconnect frequency a lot, but they still happen often. (I hope others do see 6.1 resolve the problem, and it’s just me!)

1 Like

For folks who are seeing this occur, could you share:

  1. The BIOS version you are on.
  2. The OS you are using (if on Linux, also the kernel version)
  3. What Expansion Cards and other peripherals you have plugged into each Expansion Card bay.
1 Like

For completeness sake (even though at the moment I seem to be OK):

  • 11th Gen Core-i7
  • BIOS 3.17 Beta
  • Linux michael-harvest 6.1.1-1-MANJARO #1 SMP PREEMPT_DYNAMIC Wed Dec 21 23:21:50 UTC 2022 x86_64 GNU/Linux
  • 2x USB-C (in the right- and left-rear bays)
  • 1TB storage module (in left-forward bay)
  • USB-A (in the right-forward bay)
  1. 3.10
  2. Windows 11 22H2
  3. 2x USB-A (top-left/bottom-right), 1x USB-C (top-right), 1x 256GB card (bottom-left) .

This is not a Fedora specific bug. Extended use on Windows and every linux distro I’ve tried in the past (Mint, Fedora, Ubuntu, Arch, Endeavour), whether as a boot drive or a storage drive booting off of the internal SSD, will cause random disconnects. For some reason, 6.2.0-rc2 on Fedora does not experience random disconnects for me. It’s not a bug in fedora or linux as a whole, but the new linux kernel is the first thing that has provided a fix (honestly, the bug is that it works on this kernel and nothing else). When reverting to earlier kernels, or using the drive in Windows, the problem still occurs.

Hmm, have you tried the suggestions in this post?

I’d like to get better numbers on this to see if others notice a change.

1 Like

I’m currently on the max performance with turbo setting iirc, but I will change to each of those settings and use the drive on Windows as a storage device and later booting off of it with Fedora on 6.2.0-rc2 and an older kernel to see if max non-turbo or max battery has any effect on the issue. It seems like that user still experienced the problem after changing this bios setting though.

I am also seeing random disconnects to my dock (Lenovo 40B00135US) and I also have a charge limit set on my laptop. I will have to pay attention to see if I have the same correlation. My random disconnects happen on both Arch Linux and FreeBSD 13.1. 11th Gen board on 3.17 bios.

Could you please let me know your exact kernel version? Unfortunately, the problem has come back on 6.1.5, 6.2.0-rc2, and 6.2.0-rc3 for me.

Currently running 6.1.1-1-MANJARO #1 SMP PREEMPT_DYNAMIC

Matt_HartleyFramework Team

For folks who are seeing this occur, could you share:

  1. The BIOS version you are on.
  2. The OS you are using (if on Linux, also the kernel version)
  3. What Expansion Cards and other peripherals you have plugged into each Expansion Card bay.

I keep my Dropbox on my flash drive, so it gets writes both when I use my local shared folders to add or edit files, and when my dropbox app syncs in the background. I will often get cycles of USB disconnect/reconnect chimes

  1. BIOS Version/Date INSYDE Corp. 03.06, 10/18/2021
  2. Windows 11 Home, OS build 22621.1105
  3. The configuration where I was getting daily disconnect/reconnect chime cycles:
  • USBC dongle with power delivery in top right
  • 1TB flash drive in bottom right
  • HDMI top left
  • USB-A bottom left

I tried swapping the power with the drive, so they were both still on the right but the power was below the drive. This still led to daily errors.

Now my current configuration has been running for a few days without me noticing an issue:

  • HDML in top-right
  • USB-C dongle with power delivery in bottom right
  • USB-C to USB-A hub dongle in top left
  • 1TB flash drive in bottom left

I will report back if I get an error with this config. For now, I am assuming that proximity to the power connection is a culprit, and hoping that my drive keeps staying connected if it’s on the other side of the laptop from the power input.

I am still getting the disconnect / reconnect chimes after moving the power delivery and docking bay flash drive to opposite sides of the laptop. Here is what I see in Event Viewer when the chimes happen. The D: drive is my internal 1TB expansion card flash drive. Hopefully these logs can help.

Log Name:      System
Source:        Microsoft-Windows-Ntfs
Date:          1/18/2023 11:47:43 PM
Event ID:      140
Task Category: None
Level:         Warning
Keywords:      (8)
User:          SYSTEM
Computer:      OTTOBOT
Description:
The system failed to flush data to the transaction log. Corruption may occur in VolumeId: D:, DeviceName: \Device\HarddiskVolume20.

           Failure status: A device which does not exist was specified.

           Device GUID: {66e29666-9ff2-a7a9-f923-e35f79f012dd}
           Device manufacturer:         
           Device model: USB DISK 3.2    
           Device revision: PMAP
           Device serial number: 0700199C1EA32D00
           Bus type: USB

           Adapter serial number: 

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Ntfs" Guid="{3ff37a1c-a68d-4d6e-8c9b-f79e8b16c482}" />
    <EventID>140</EventID>
    <Version>1</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000008</Keywords>
    <TimeCreated SystemTime="2023-01-19T07:47:43.7265946Z" />
    <EventRecordID>24718</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="6048" />
    <Channel>System</Channel>
    <Computer>OTTOBOT</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="VolumeIdLength">2</Data>
    <Data Name="VolumeId">D:</Data>
    <Data Name="DeviceNameLength">24</Data>
    <Data Name="DeviceName">\Device\HarddiskVolume20</Data>
    <Data Name="Error">0xc000000e</Data>
    <Data Name="DeviceGuid">{66e29666-9ff2-a7a9-f923-e35f79f012dd}</Data>
    <Data Name="VendorIdLength">8</Data>
    <Data Name="VendorId">
    </Data>
    <Data Name="ProductIdLength">16</Data>
    <Data Name="ProductId">USB DISK 3.2    </Data>
    <Data Name="ProductRevisionLength">4</Data>
    <Data Name="ProductRevision">PMAP</Data>
    <Data Name="DeviceSerialNumberLength">16</Data>
    <Data Name="DeviceSerialNumber">0700199C1EA32D00</Data>
    <Data Name="BusType">7</Data>
    <Data Name="AdapterSerialNumberLength">0</Data>
    <Data Name="AdapterSerialNumber">
    </Data>
  </EventData>
</Event>

Log Name:      System
Source:        disk
Date:          1/18/2023 11:47:43 PM
Event ID:      51
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      OTTOBOT
Description:
An error was detected on device \Device\Harddisk1\DR16 during a paging operation.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="disk" />
    <EventID Qualifiers="32772">51</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2023-01-19T07:47:43.7264803Z" />
    <EventRecordID>24717</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="6560" />
    <Channel>System</Channel>
    <Computer>OTTOBOT</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\Harddisk1\DR16</Data>
    <Binary>040080000100000000000000330004802D0100000E0000C0000000000000000000000000000000005C19130000000000FFFFFFFF0100000058000008000000001C200A1282032040001000000F0000000020FCDB03CEFFFF1814DBE303CEFFFF0000000000000000B0C6F1F203CEFFFF000000000000000098345D00000000002A00005D349800000800000000000000000000000000000000000000000000000000000000000000</Binary>
  </EventData>
</Event>

Log Name:      System
Source:        disk
Date:          1/18/2023 11:47:43 PM
Event ID:      51
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      OTTOBOT
Description:
An error was detected on device \Device\Harddisk1\DR16 during a paging operation.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="disk" />
    <EventID Qualifiers="32772">51</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2023-01-19T07:47:43.7264803Z" />
    <EventRecordID>24716</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="15020" />
    <Channel>System</Channel>
    <Computer>OTTOBOT</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\Harddisk1\DR16</Data>
    <Binary>040080000100000000000000330004802D0100000E0000C0000000000000000000000000000000005C19130000000000FFFFFFFF0100000058000008000000001C200A1282032040001000000F000000006076EA03CEFFFF1814DBE303CEFFFF0000000000000000B0C6F1F203CEFFFF0000000000000000A0345D00000000002A00005D34A000000800000000000000000000000000000000000000000000000000000000000000</Binary>
  </EventData>
</Event>

Log Name:      System
Source:        disk
Date:          1/18/2023 11:47:43 PM
Event ID:      51
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      OTTOBOT
Description:
An error was detected on device \Device\Harddisk1\DR16 during a paging operation.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="disk" />
    <EventID Qualifiers="32772">51</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2023-01-19T07:47:43.7264803Z" />
    <EventRecordID>24715</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="15020" />
    <Channel>System</Channel>
    <Computer>OTTOBOT</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\Harddisk1\DR16</Data>
    <Binary>040080000100000000000000330004802D0100000E0000C0000000000000000000000000000000005C19130000000000FFFFFFFF0100000058000008000000001C200A1282032040001000000F00000000E035E203CEFFFF1814DBE303CEFFFF0000000000000000B0C6F1F203CEFFFF000000000000000008C15E00000000002A00005EC10800000800000000000000000000000000000000000000000000000000000000000000</Binary>
  </EventData>
</Event>

Log Name:      System
Source:        disk
Date:          1/18/2023 11:47:43 PM
Event ID:      51
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      OTTOBOT
Description:
An error was detected on device \Device\Harddisk1\DR16 during a paging operation.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="disk" />
    <EventID Qualifiers="32772">51</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2023-01-19T07:47:43.7254702Z" />
    <EventRecordID>24714</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="15020" />
    <Channel>System</Channel>
    <Computer>OTTOBOT</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\Harddisk1\DR16</Data>
    <Binary>040180000100000000000000330004802D0100000E0000C0000000000000000000000000000000005C19130000000000FFFFFFFF0100000058000008000000001C200A1282032040001000000F00000000F00FE203CEFFFF78E98EDB03CEFFFF000000000000000060F3F0F203CEFFFF000000000000000008C15E00000000002A00005EC10800000800000000000000000000000000000000000000000000000000000000000000</Binary>
  </EventData>
</Event>

Further events which are correlated with the chime sound:

Log Name:      System
Source:        nhi
Date:          1/19/2023 3:12:48 AM
Event ID:      9008
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OTTOBOT
Description:
Driver exit RTD3.
 All the connected devices will now cause DeviceConnected events.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="nhi" />
    <EventID Qualifiers="16388">9008</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2023-01-19T11:12:48.0295709Z" />
    <EventRecordID>24918</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="18604" />
    <Channel>System</Channel>
    <Computer>OTTOBOT</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Binary>00000000010000000000000030230440000000000000000000000000000000000000000000000000</Binary>
  </EventData>
</Event>

Screenshot_20230119_031336

For Linux users, please look into the results of this immediately after an event and once you’re able to get logged in:

journalctl --since "10 minutes ago"

Would be interesting to see what we see there in terms of what may be happening.

After weeks of not really having an issue, I just had what I might call an extreme version of it.

Specifically, I went to reboot the machine, and suddenly, the expansion drive was not visible even to the BIOS!!!

As far as I can tell the only answer to “how did I fix it” was “voodoo”. I threatened it with a dead chicken and the next time it rebooted, the drive was visible again.

This is the first time I can think of, though, where the drive refused to connect right from the start.

This problem certainly reaches the level of mysticism for me. For instance, here’s a setup I’ve found where the drive never disconnects (at least for the last month of connecting it this way):

However, when I try a shorter USB-C cable, it disconnects all of the time. In fact, when it disconnected yesterday, it wouldn’t automatically reconnect. Instead there were the following errors in the Event Log and Device Manager, and I had to reboot to fix it:

Event Log Error: Windows failed to start the USB xHCI Compliant Host Controller for the following reason: Controller reset timed out. Check with your computer manufacturer for an updated firmware for the controller.

Device Manager Error on the first of the two “Intel (R) USB 3.10 eXtensible Host Controller - 1.20 (Microsoft)” Universal Serial Bus Controllers: Windows has stopped this device because it has reported problems (Code 43).

I have a USB-A drive which disconnects frequently on the right hand side USB-A port (like when I’m backing up the computer). The left port seems more solid, except the only way to get the drive to show up in the left USB-A port is to plug it in halfway, wait for Windows to recognize it, and then I can plug it in all of the way. This has nothing to do with the USB-A cards, as I can swap them around without changing how each side behaves.

I do a little experimenting every day. For instance, now I’m attempting to plug the USB-A drive in on the right hand side halfway (even though I don’t need to), and it seems more stable. Of course, given how flaky the patterns are, I could find any number of patterns involving what shirt I wear or dances I perform while plugging devices in.

I have already replaced the mainboard and the 1 TB card and sent the old ones back to Framework. I don’t know if they’ve been able to reproduce the problems on their end, but it would be interesting to know if they couldn’t reproduce the disconnect on a Windows 10 machine using my old Mainboard and 1 TB card. If they can’t, then maybe it’s purely a software problem (or something to do with the negative vibes in my apartment).

Some new anecdata:

For the most part, once I’m booted off of /dev/sda (aka the 1TB expansion module), it works without a hitch.

Occasionally, though—usually after I’ve done an update, but sometimes if I’m rebooting for other reasons—the Framework (11th Gen Core i7, latest BIOS) refuses to see the module, right from the get go. BIOS doesn’t see it as a device, let alone a bootable drive.

The most recent time that happened, simply popping the module out and re-seating made no difference. It was stubbornly being ignored. I wound up swapping it out of the front-left bay to the back-left bay, mostly on a lark.

That worked.

Of course, it could be coincidence. One way or another, there is DEFINITELY something weird and I’m pretty sure it’s a hardware problem, or else a deep firmware problem.

I’m surprised this hasn’t been brought up here yet, however this could be due to the card not responding correctly or fast enough to UAS commands on Linux (not sure about Windows). For Linux specifically, this may be solved with the following added to the boot command line in grub (GRUB_CMDLINE_LINUX or GRUB_CMDLINE_LINUX_DEFUALT):

usb-storage.quirks=13fe:6500:u

That will disable UAS, which may sacrifice a bit of speed but should be much more stable. I personally have random disconnect issues on various machines with various external NVME adapters and this fixes it for all of them.

Note that this is specifically for the card the OP used as the usb vendor and product ID was pulled from the dmesg output they provided. If this is needed for another adapter/card you may need to change the “13fe:6500” part with the information from dmesg that looks like this:

[<    0.017413>] usb 2-2: New USB device found, idVendor=13fe, idProduct=6500, bcdDevice= 1.10
4 Likes

As I understand it, this is a kernel-level issue. As described above, I occasionally “lose sight” of the drive at the BIOS level, suggesting that it is not an operating system issue at all!