[RESPONDED] 1TB expansion card disconnects randomly

Be_Far · December 31, 2022, 4:21am

@Michael_Scot_Shappe just to confirm, this is when booting off of the card? If so I’ll update my Fedora install to 6.1 right away. A few days is much better than the hour-or-so I’ve been getting.

Michael_Scot_Shappe · December 31, 2022, 4:34am

Correct. I’m running Manjaro and as soon as they offered a stable 6.1 kernel I took it. No disconnects all week, including a chunk of time running less well ventilated than it usually does at home (and hence running a little hotter; I’ve been traveling). But I’ve been using it as my daily work driver even so, so it’s been running solidly for 8+ hours a day.

Since we never really nailed down what was causing it, it’s hard to say that it’s fixed; only that it hasn’t happened since I switched to a 6.1 kernel.

Be_Far · January 4, 2023, 4:34am

Update: I’ve been fiddling with nvidia module patches on Fedora 37 running kernel 6.2.0-rc2 from the Rawhide repo for three hours now and have not encountered a single random disconnect! For context, I had two in the 30 minutes it took me to upgrade from 6.0.15 kernel to the new one (and one while I was back on 6.0.15 reverting my failed tweaks).

This is very silly, and I have no idea what might have changed in a single minor release that would suddenly fix a relatively obscure boot device’s constant disconnects.

And now, we wait for FC38 to release, because my hack to compile the nvidia kernel module on 6.2 failed…

Michael_Scot_Shappe · January 4, 2023, 8:27pm

At this point, the only oddities I"m still seeing on a 6.1 kernel are with hibernate or sleep not coming back right. Everything else seems to be happy right now.

Matt_Hartley · January 9, 2023, 7:07pm

Are we feeling like 6.1 and 6.2.0-rc2 have resolved the disconnect issue? Been spending this morning tracking down Fedora bugs, I’d like to call this one solved. Please confirm - thanks.

Michael_Scot_Shappe · January 9, 2023, 7:22pm

Right now, I can give you absence of evidence: I have not seen a disconnect that broke my system since switching to 6.1 stable.

Of course, that means I can’t prove evidence of absence, that it’s absolutely fixed! Nobody else seems to have chimed in, so I’d love to see more evidence from people who have been plagued by the issue on Linux!

Matti · January 9, 2023, 9:24pm

I installed 6.1 as soon as I saw this thread, rebooted into it, and my SD card disconnected again within half an hour.

My system is an 11th gen mainboard, BIOS 3.17 (the one that just hit LVFS), KUbuntu 22.04, kernel 6.1.4:

$ uname -r
6.1.4-060104-generic

The BIOS settings I have that might be relevant: the battery is capped at 80%, and I have this:

bios -> Advanced -> Boot performance mode = Max Non-Turbo Performance

thanks to this comment: 1TB expansion card disconnects randomly - #12 by Shy_Guy . Setting that a few months ago reduced the disconnect frequency a lot, but they still happen often. (I hope others do see 6.1 resolve the problem, and it’s just me!)

Matt_Hartley · January 10, 2023, 7:28pm

For folks who are seeing this occur, could you share:

The BIOS version you are on.
The OS you are using (if on Linux, also the kernel version)
What Expansion Cards and other peripherals you have plugged into each Expansion Card bay.

Michael_Scot_Shappe · January 10, 2023, 7:49pm

For completeness sake (even though at the moment I seem to be OK):

11th Gen Core-i7
BIOS 3.17 Beta
Linux michael-harvest 6.1.1-1-MANJARO #1 SMP PREEMPT_DYNAMIC Wed Dec 21 23:21:50 UTC 2022 x86_64 GNU/Linux
2x USB-C (in the right- and left-rear bays)
1TB storage module (in left-forward bay)
USB-A (in the right-forward bay)

amoun · January 10, 2023, 8:06pm

Well mine is a 256GB card.
11Gen Batch 8 Feb 2022

BIOS 3.17 Seems to be getting worse, was monthly then fortnightly now sometimes 2 or three times a week.

Pre-built: Win 10 > Win 11 on arrival. Ubuntu 22.04 to 22.10 on expansion card.

Nearly always plugged in so little info on battery.
Use Win 99% of the time so no data on Ubuntu

Ports 2 x USB C 1 x Micro SD 1 x 256 card

Framework mains supply, battery max permanently at 78%

No peripherals.

Fan hardly ever comes on: temp 40 to 50

john_doe · January 10, 2023, 9:56pm

3.10
Windows 11 22H2
2x USB-A (top-left/bottom-right), 1x USB-C (top-right), 1x 256GB card (bottom-left) .

Be_Far · January 12, 2023, 4:36pm

This is not a Fedora specific bug. Extended use on Windows and every linux distro I’ve tried in the past (Mint, Fedora, Ubuntu, Arch, Endeavour), whether as a boot drive or a storage drive booting off of the internal SSD, will cause random disconnects. For some reason, 6.2.0-rc2 on Fedora does not experience random disconnects for me. It’s not a bug in fedora or linux as a whole, but the new linux kernel is the first thing that has provided a fix (honestly, the bug is that it works on this kernel and nothing else). When reverting to earlier kernels, or using the drive in Windows, the problem still occurs.

Matt_Hartley · January 13, 2023, 12:51am

Hmm, have you tried the suggestions in this post?

I’d like to get better numbers on this to see if others notice a change.

Be_Far · January 13, 2023, 4:07pm

I’m currently on the max performance with turbo setting iirc, but I will change to each of those settings and use the drive on Windows as a storage device and later booting off of it with Fedora on 6.2.0-rc2 and an older kernel to see if max non-turbo or max battery has any effect on the issue. It seems like that user still experienced the problem after changing this bios setting though.

Tom1 · January 13, 2023, 6:59pm

I am also seeing random disconnects to my dock (Lenovo 40B00135US) and I also have a charge limit set on my laptop. I will have to pay attention to see if I have the same correlation. My random disconnects happen on both Arch Linux and FreeBSD 13.1. 11th Gen board on 3.17 bios.

Be_Far · January 16, 2023, 7:02pm

Could you please let me know your exact kernel version? Unfortunately, the problem has come back on 6.1.5, 6.2.0-rc2, and 6.2.0-rc3 for me.

Michael_Scot_Shappe · January 16, 2023, 7:29pm

Currently running 6.1.1-1-MANJARO #1 SMP PREEMPT_DYNAMIC

Victor_Andersen · January 16, 2023, 10:22pm

Matt_Hartley Framework Team

For folks who are seeing this occur, could you share:

The BIOS version you are on.

The OS you are using (if on Linux, also the kernel version)

What Expansion Cards and other peripherals you have plugged into each Expansion Card bay.

I keep my Dropbox on my flash drive, so it gets writes both when I use my local shared folders to add or edit files, and when my dropbox app syncs in the background. I will often get cycles of USB disconnect/reconnect chimes

BIOS Version/Date INSYDE Corp. 03.06, 10/18/2021
Windows 11 Home, OS build 22621.1105
The configuration where I was getting daily disconnect/reconnect chime cycles:

USBC dongle with power delivery in top right
1TB flash drive in bottom right
HDMI top left
USB-A bottom left

I tried swapping the power with the drive, so they were both still on the right but the power was below the drive. This still led to daily errors.

Now my current configuration has been running for a few days without me noticing an issue:

HDML in top-right
USB-C dongle with power delivery in bottom right
USB-C to USB-A hub dongle in top left
1TB flash drive in bottom left

I will report back if I get an error with this config. For now, I am assuming that proximity to the power connection is a culprit, and hoping that my drive keeps staying connected if it’s on the other side of the laptop from the power input.

Victor_Andersen · January 19, 2023, 9:33am

I am still getting the disconnect / reconnect chimes after moving the power delivery and docking bay flash drive to opposite sides of the laptop. Here is what I see in Event Viewer when the chimes happen. The D: drive is my internal 1TB expansion card flash drive. Hopefully these logs can help.

Log Name:      System
Source:        Microsoft-Windows-Ntfs
Date:          1/18/2023 11:47:43 PM
Event ID:      140
Task Category: None
Level:         Warning
Keywords:      (8)
User:          SYSTEM
Computer:      OTTOBOT
Description:
The system failed to flush data to the transaction log. Corruption may occur in VolumeId: D:, DeviceName: \Device\HarddiskVolume20.

           Failure status: A device which does not exist was specified.

           Device GUID: {66e29666-9ff2-a7a9-f923-e35f79f012dd}
           Device manufacturer:         
           Device model: USB DISK 3.2    
           Device revision: PMAP
           Device serial number: 0700199C1EA32D00
           Bus type: USB

           Adapter serial number: 

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Ntfs" Guid="{3ff37a1c-a68d-4d6e-8c9b-f79e8b16c482}" />
    <EventID>140</EventID>
    <Version>1</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000008</Keywords>
    <TimeCreated SystemTime="2023-01-19T07:47:43.7265946Z" />
    <EventRecordID>24718</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="6048" />
    <Channel>System</Channel>
    <Computer>OTTOBOT</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="VolumeIdLength">2</Data>
    <Data Name="VolumeId">D:</Data>
    <Data Name="DeviceNameLength">24</Data>
    <Data Name="DeviceName">\Device\HarddiskVolume20</Data>
    <Data Name="Error">0xc000000e</Data>
    <Data Name="DeviceGuid">{66e29666-9ff2-a7a9-f923-e35f79f012dd}</Data>
    <Data Name="VendorIdLength">8</Data>
    <Data Name="VendorId">
    </Data>
    <Data Name="ProductIdLength">16</Data>
    <Data Name="ProductId">USB DISK 3.2    </Data>
    <Data Name="ProductRevisionLength">4</Data>
    <Data Name="ProductRevision">PMAP</Data>
    <Data Name="DeviceSerialNumberLength">16</Data>
    <Data Name="DeviceSerialNumber">0700199C1EA32D00</Data>
    <Data Name="BusType">7</Data>
    <Data Name="AdapterSerialNumberLength">0</Data>
    <Data Name="AdapterSerialNumber">
    </Data>
  </EventData>
</Event>

Log Name:      System
Source:        disk
Date:          1/18/2023 11:47:43 PM
Event ID:      51
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      OTTOBOT
Description:
An error was detected on device \Device\Harddisk1\DR16 during a paging operation.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="disk" />
    <EventID Qualifiers="32772">51</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2023-01-19T07:47:43.7264803Z" />
    <EventRecordID>24717</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="6560" />
    <Channel>System</Channel>
    <Computer>OTTOBOT</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\Harddisk1\DR16</Data>
    <Binary>040080000100000000000000330004802D0100000E0000C0000000000000000000000000000000005C19130000000000FFFFFFFF0100000058000008000000001C200A1282032040001000000F0000000020FCDB03CEFFFF1814DBE303CEFFFF0000000000000000B0C6F1F203CEFFFF000000000000000098345D00000000002A00005D349800000800000000000000000000000000000000000000000000000000000000000000</Binary>
  </EventData>
</Event>

Log Name:      System
Source:        disk
Date:          1/18/2023 11:47:43 PM
Event ID:      51
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      OTTOBOT
Description:
An error was detected on device \Device\Harddisk1\DR16 during a paging operation.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="disk" />
    <EventID Qualifiers="32772">51</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2023-01-19T07:47:43.7264803Z" />
    <EventRecordID>24716</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="15020" />
    <Channel>System</Channel>
    <Computer>OTTOBOT</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\Harddisk1\DR16</Data>
    <Binary>040080000100000000000000330004802D0100000E0000C0000000000000000000000000000000005C19130000000000FFFFFFFF0100000058000008000000001C200A1282032040001000000F000000006076EA03CEFFFF1814DBE303CEFFFF0000000000000000B0C6F1F203CEFFFF0000000000000000A0345D00000000002A00005D34A000000800000000000000000000000000000000000000000000000000000000000000</Binary>
  </EventData>
</Event>

Log Name:      System
Source:        disk
Date:          1/18/2023 11:47:43 PM
Event ID:      51
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      OTTOBOT
Description:
An error was detected on device \Device\Harddisk1\DR16 during a paging operation.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="disk" />
    <EventID Qualifiers="32772">51</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2023-01-19T07:47:43.7264803Z" />
    <EventRecordID>24715</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="15020" />
    <Channel>System</Channel>
    <Computer>OTTOBOT</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\Harddisk1\DR16</Data>
    <Binary>040080000100000000000000330004802D0100000E0000C0000000000000000000000000000000005C19130000000000FFFFFFFF0100000058000008000000001C200A1282032040001000000F00000000E035E203CEFFFF1814DBE303CEFFFF0000000000000000B0C6F1F203CEFFFF000000000000000008C15E00000000002A00005EC10800000800000000000000000000000000000000000000000000000000000000000000</Binary>
  </EventData>
</Event>

Log Name:      System
Source:        disk
Date:          1/18/2023 11:47:43 PM
Event ID:      51
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      OTTOBOT
Description:
An error was detected on device \Device\Harddisk1\DR16 during a paging operation.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="disk" />
    <EventID Qualifiers="32772">51</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2023-01-19T07:47:43.7254702Z" />
    <EventRecordID>24714</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="15020" />
    <Channel>System</Channel>
    <Computer>OTTOBOT</Computer>
    <Security />
  </System>
  <EventData>
    <Data>\Device\Harddisk1\DR16</Data>
    <Binary>040180000100000000000000330004802D0100000E0000C0000000000000000000000000000000005C19130000000000FFFFFFFF0100000058000008000000001C200A1282032040001000000F00000000F00FE203CEFFFF78E98EDB03CEFFFF000000000000000060F3F0F203CEFFFF000000000000000008C15E00000000002A00005EC10800000800000000000000000000000000000000000000000000000000000000000000</Binary>
  </EventData>
</Event>

Further events which are correlated with the chime sound:

Log Name:      System
Source:        nhi
Date:          1/19/2023 3:12:48 AM
Event ID:      9008
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      OTTOBOT
Description:
Driver exit RTD3.
 All the connected devices will now cause DeviceConnected events.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="nhi" />
    <EventID Qualifiers="16388">9008</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2023-01-19T11:12:48.0295709Z" />
    <EventRecordID>24918</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="18604" />
    <Channel>System</Channel>
    <Computer>OTTOBOT</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Binary>00000000010000000000000030230440000000000000000000000000000000000000000000000000</Binary>
  </EventData>
</Event>

Screenshot_20230119_031336

Matt_Hartley · January 19, 2023, 7:14pm

For Linux users, please look into the results of this immediately after an event and once you’re able to get logged in:

journalctl --since "10 minutes ago"

Would be interesting to see what we see there in terms of what may be happening.