Success in recovering from bad BIOS upgrade - Framework 13 AMD 7040

Community Support
,
1

On my Framework 13 AMD, last week I attempted updating from 3.05 to 3.07. I initiated the update via fwupdmgr on Ubuntu 24.04. After the system rebooted, it started showing the update progress, as expected. But then about ~10 minutes in, the screen glitched out and ended up stuck in a state like shown here:

BIOS update glitch
BIOS update glitch1920×1440 181 KB

I could still hear the fans spooling up and down, so I waited, but eventually the fans spun all the way down, and the laptop stayed there for hours. Nothing would respond to user input, even holding down the power button. I ended up just letting the battery die. Note there have been a few other reports of this in the community forum, including updates to older BIOS versions.

Upon powering it back up, I got nothing on the display. The diagnostics LEDs would flash a sequence of WGGGGGGGGGGGGOGBGBBGGG.

I went back and forth with support for a couple days (DRAM shuffle, mainboard reset, try external monitor, etc), but eventually they said that I’d need to purchase a new motherboard (since I was out of warranty). So, with nothing else to lose…

I didn’t have a SPI flash programmer, so I ended up soldering some 30 gauge wire to the SPI flash pins, ran that to a level shifter (with the flash side powered by an external power supply), then to a Raspberry Pi. I first dumped the flash (a few times, to be sure I had a decent connection), then extracted the ROM contents from the 3.07 UEFI capsule file, and wrote that out to flash. You can see my (super hacky) setup here:

super hacky SPI flasher
super hacky SPI flasher1920×1440 380 KB

And… it worked! Happy to have saved $500 on a new motherboard!

Just wanted to throw this out there for anyone else who may run into the same issue. Also - please don’t try this unless you’ve exhausted all options with support!

8 Likes
2

I heard that you can’t downgrade from 3.07 to 3.05 by normal means, maybe this super hacky way is the way

3

Bit more detail here:

In my picture above, I have an external big adjustable DC power supply (the type you’d see on an electronics lab bench) set at 1.9V. You’ll want to make sure your battery is disconnected before attempting any of this. When powering on the DC supply, I noticed it was drawing 0.2A, which is much more than the Winbond flash part should be consuming, even while actively being read/written. This implies that (maybe?) I was powering other parts of the 1.8V logic on the mainboard, but it didn’t cause a problem for me, and I left it powered for several hours non-stop. I also had the /RESET and /WP pins of the SPI flash tied high (to the 1.9V of my power supply). I did not bother connecting these to the Raspberry PI for it to control.

The level-shifter I used was this one from Amazon, which I just happened to have on-hand. I expect anything capable of adapting from 3.3V to 1.8V at a reasonable frequency would work fine. I ran my SPI reads/writes at 256kHz - on the low side, which was important considering I was using some fairly long jumper wires.

I used this article as a great reference on how to hook things up to the Raspberry Pi and configure it to use the SPI-0 peripheral. I was using a Raspberry Pi 5, but I know that at least the Raspberry Pi 4 has the same pinout and should work identically.

After I had the SPI bus configured on the RPI (per the guide above) and had everything hooked up with the level shifter, I ran the following to confirm I was detecting the expected flash chip:

$ flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=256 --wp-status
flashrom v1.6.0-devel (git:v1.5.0-33-gc3b89597) on Linux 6.6.74+rpt-rpi-2712 (aarch64)
flashrom is free software, get the source code at https://flashrom.org

Found Winbond flash chip "W25Q256JW" (32768 kB, SPI) on linux_spi.
Protection range: start=0x00000000 length=0x00000000 (none)
Protection mode: disabled

The chip isn’t technically a W25Q256JW - it is W25R256JWEQ. But that didn’t matter - I didn’t dig in much but I expect the difference is due to the logic level and the package.

I then did a couple different reads to separate files, and made sure they matched. An example read:

$ flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=256 -r fw-bios-orig-1.bin --progress
flashrom v1.6.0-devel (git:v1.5.0-33-gc3b89597) on Linux 6.6.74+rpt-rpi-2712 (aarch64)
flashrom is free software, get the source code at https://flashrom.org

Found Winbond flash chip "W25Q256JW" (32768 kB, SPI) on linux_spi.
Reading flash... 
[READ: 100%]...done.

Next step was to extract the portion of the UBI capsule file (e.g. Framework_Laptop_13_Ryzen7040_capsule_signed_allsku_3.07.cap) that contained the actual SPI flash contents. This was my first time dealing with these UEFI capsules, but by using UEFITool, binwalk, and some basic hex comparisons (both on my dumped SPI flash contents and the capsule file from the Framework upgrade zip), I came to the following conclusions:

  1. My SPI flash seemed to have been mostly (if not fully) updated to 3.07. All of the firmware regions matched perfectly with the UEFI capsule file. The differences were only in some non-zeroed padding regions, along with the expected differences in the nvram VSS store region (where BIOS settings and other non-volatile status is saved). I still have no idea which of these two differences was causing it to fail to boot.
  2. The UEFI capsule file had some header stuff, then a couple Microsoft PE-formatted executables, then the actual SPI flash payload, then finally some trailing random data. I didn’t bother digging it into any more than necessary to figure out which region of this file I needed to flash. I primarily used UEFITool to figure out the relative offsets of the regions between my SPI flash dump and the UEFI capsule file.

Once I had my offset and length figured out, I extracted the SPI flash portion of the UEFI capsule as follows:

$ md5sum Framework_Laptop_13_Ryzen7040_capsule_signed_allsku_3.07.cap
ae4043c17d95f36f14f342e6deca3c5e  Framework_Laptop_13_Ryzen7040_capsule_signed_allsku_3.07.cap
$ tail -c +1993293 Framework_Laptop_13_Ryzen7040_capsule_signed_allsku_3.07.cap | head -c 33554432 > framework-3.07-rom.bin 
$ md5sum framework-3.07-rom.bin 
e8241cda9edf65627d84303f9e6f91af  framework-3.07-rom.bin

IMPORTANT NOTE: The offset and md5sums will be different if you’re trying to flash a different version of the BIOS.

Next, I went ahead and flashed away:

$ flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=256 -w framework-3.07-rom.bin --progress 
flashrom v1.6.0-devel (git:v1.5.0-33-gc3b89597) on Linux 6.6.74+rpt-rpi-2712 (aarch64)
flashrom is free software, get the source code at https://flashrom.org

Found Winbond flash chip "W25Q256JW" (32768 kB, SPI) on linux_spi.
Reading old flash chip contents... 
[READ: 100%]...done.
Updating flash chip contents... 
[WRITE: 100%]...Erase/write done from 0 to 1ffffff
Verifying flash... 
[READ: 100%]...VERIFIED.

I then disconnected all my wires, cleaned off the solder flux, booted up my laptop, and breathed a huge sigh of relief when I got my LUKS password prompt. Everything has been working flawlessly for the ~6 hours since doing this.

Also note this will wipe all BIOS settings to default (which I absolutely wanted to do).

After doing all this, I am noticing that fwupdmgr is reporting that it wants to update my UEFI dbx…

$ fwupdmgr get-updates
Devices with no available firmware updates: 
 • WD BLACK SN850X 2000GB
Devices with the latest available firmware version:
 • Fingerprint Sensor
 • System Firmware
Framework Laptop 13 (AMD Ryzen 7040Series)
│
└─UEFI dbx:
  │   Device ID:          362301da643102b9f38477387e2193e57abaa590
  │   Summary:            UEFI revocation database
  │   Current version:    20230501
  │   Minimum Version:    20230501
  │   Vendor:             UEFI:Microsoft
  │   Install Duration:   1 second
  │   GUIDs:              f8ba2887-9411-5c36-9cee-88995bb39731 ← UEFI\CRT_A1117F516A32CEFCBA3F2D1ACE10A87972FD6BBE8FE0D0B996E09E65D802A503&ARCH_X64
  │                       115f7cac-f705-5d34-9a47-37177c3e8514 ← UEFI\CRT_B38FAD316F525F27B27A21B486456C3E4279748BF16893827BF16FE659C0F75E&ARCH_X64
  │   Device Flags:       • Internal device
  │                       • Updatable
  │                       • Supported on remote server
  │                       • Needs a reboot after installation
  │                       • Device is usable for the duration of the update
  │                       • Only version upgrades are allowed
  │                       • Signed Payload
  │ 
  └─Secure Boot dbx Configuration Update:
        New version:      20241101
        Remote ID:        lvfs
        Release ID:       105821
        Summary:          UEFI Secure Boot Forbidden Signature Database
        Variant:          x64
        License:          Proprietary
        Size:             15.1 kB
        Created:          2025-01-17
        Urgency:          High
        Tested by Linux Foundation:
          Tested:         2025-01-20
          Distribution:   fedora 41 (workstation)
          Old version:    20240301
          Version[fwupd]: 2.0.4
        Vendor:           Linux Foundation
        Duration:         1 second
        Release Flags:    • Trusted metadata
                          • Is upgrade
                          • Tested by trusted vendor
        Description:      
        This updates the list of forbidden signatures (the "dbx") to the latest release from Microsoft.
        
        An insecure version of Howyar's SysReturn software was added, due to a security vulnerability that allowed an attacker to bypass UEFI Secure Boot.
        Issues:           529659
                          CVE-2024-7344
        Checksum:         d661d4a0aaca09dfa9e56967ca2467b0575fc07cb704d182fa8c68225452957f

From my very limited experience on UEFI (from going through this recovery process), I learned that is a section in flash with a list of signatures of firmware that should be “blacklisted” for secure-boot purposes. I don’t use secure-boot (shame on me), and considering my shotty experience with the Framework BIOS update process, I’m not touching this with a 10-foot pole. I’ll wait to do anything that writes to this SPI flash until absolutely necessary.

Happy hacking!

7 Likes
4

So, this does show how risky a BIOS upgrade is on the FW13 AMD.

If the BIOS update fails, the motherboard is bricked, unless you use some of the soldering iron based recovery process described above.

Maybe on future motherboards, FW might make it slightly more support friendly.
I.e. make BIOS updates “un-brickable”.

At least we know what “WGGGGGGGGGGGGOGBGBBGGG” means now.
BIOS Code: 00011010 or 0x1A
It means that the BIOS is probably corrupted.

it would have been really interesting to see what the EC console output during this fault.

5

Amazing work and thank you for documenting this.

However, I’m surprised and a little concerned support concluded you’d need to purchase a new motherboard. Excluding any obvious “customer induced damage”, this seems like a software issue. In an ideal world, it’d be RMA’ed for further diagnosis regardless of warranty status (the AMD 7040 series is not that old - began shipping in Oct 2023?).

6

Fully agree! I replied to the support email thread yesterday after I got this all working, explained what I did, plus offered to provide them my original/broken flash dump. They responded and simply advised me to “keep on monitoring the device”. Pretty sure I didn’t get past the first-level support there…

If any Framework engineers see this, offer still stands! I’m more than happy to provide the dump and would very much appreciate someone looking into what may have caused this.

1 Like
7

For what it’s worth, dbx is stored in NVRAM and updated the same way as all other UEFI variables stored in NVRAM. There is no Framework-specific reflashing logic used in updating it. It is not stored in a special flash region.

1 Like