Success in recovering from bad BIOS upgrade - Framework 13 AMD 7040

Community Support
,
1

On my Framework 13 AMD, last week I attempted updating from 3.05 to 3.07. I initiated the update via fwupdmgr on Ubuntu 24.04. After the system rebooted, it started showing the update progress, as expected. But then about ~10 minutes in, the screen glitched out and ended up stuck in a state like shown here:

BIOS update glitch
BIOS update glitch1920×1440 181 KB

I could still hear the fans spooling up and down, so I waited, but eventually the fans spun all the way down, and the laptop stayed there for hours. Nothing would respond to user input, even holding down the power button. I ended up just letting the battery die. Note there have been a few other reports of this in the community forum, including updates to older BIOS versions.

Upon powering it back up, I got nothing on the display. The diagnostics LEDs would flash a sequence of WGGGGGGGGGGGGOGBGBBGGG.

I went back and forth with support for a couple days (DRAM shuffle, mainboard reset, try external monitor, etc), but eventually they said that I’d need to purchase a new motherboard (since I was out of warranty). So, with nothing else to lose…

I didn’t have a SPI flash programmer, so I ended up soldering some 30 gauge wire to the SPI flash pins, ran that to a level shifter (with the flash side powered by an external power supply), then to a Raspberry Pi. I first dumped the flash (a few times, to be sure I had a decent connection), then extracted the ROM contents from the 3.07 UEFI capsule file, and wrote that out to flash. You can see my (super hacky) setup here:

super hacky SPI flasher
super hacky SPI flasher1920×1440 380 KB

And… it worked! Happy to have saved $500 on a new motherboard!

Just wanted to throw this out there for anyone else who may run into the same issue. Also - please don’t try this unless you’ve exhausted all options with support!

17 Likes
2

I heard that you can’t downgrade from 3.07 to 3.05 by normal means, maybe this super hacky way is the way

3

Bit more detail here:

In my picture above, I have an external big adjustable DC power supply (the type you’d see on an electronics lab bench) set at 1.9V. You’ll want to make sure your battery is disconnected before attempting any of this. When powering on the DC supply, I noticed it was drawing 0.2A, which is much more than the Winbond flash part should be consuming, even while actively being read/written. This implies that (maybe?) I was powering other parts of the 1.8V logic on the mainboard, but it didn’t cause a problem for me, and I left it powered for several hours non-stop. I also had the /RESET and /WP pins of the SPI flash tied high (to the 1.9V of my power supply). I did not bother connecting these to the Raspberry PI for it to control.

The level-shifter I used was this one from Amazon, which I just happened to have on-hand. I expect anything capable of adapting from 3.3V to 1.8V at a reasonable frequency would work fine. I ran my SPI reads/writes at 256kHz - on the low side, which was important considering I was using some fairly long jumper wires.

I used this article as a great reference on how to hook things up to the Raspberry Pi and configure it to use the SPI-0 peripheral. I was using a Raspberry Pi 5, but I know that at least the Raspberry Pi 4 has the same pinout and should work identically.

After I had the SPI bus configured on the RPI (per the guide above) and had everything hooked up with the level shifter, I ran the following to confirm I was detecting the expected flash chip:

$ flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=256 --wp-status
flashrom v1.6.0-devel (git:v1.5.0-33-gc3b89597) on Linux 6.6.74+rpt-rpi-2712 (aarch64)
flashrom is free software, get the source code at https://flashrom.org

Found Winbond flash chip "W25Q256JW" (32768 kB, SPI) on linux_spi.
Protection range: start=0x00000000 length=0x00000000 (none)
Protection mode: disabled

The chip isn’t technically a W25Q256JW - it is W25R256JWEQ. But that didn’t matter - I didn’t dig in much but I expect the difference is due to the logic level and the package.

I then did a couple different reads to separate files, and made sure they matched. An example read:

$ flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=256 -r fw-bios-orig-1.bin --progress
flashrom v1.6.0-devel (git:v1.5.0-33-gc3b89597) on Linux 6.6.74+rpt-rpi-2712 (aarch64)
flashrom is free software, get the source code at https://flashrom.org

Found Winbond flash chip "W25Q256JW" (32768 kB, SPI) on linux_spi.
Reading flash... 
[READ: 100%]...done.

Next step was to extract the portion of the UBI capsule file (e.g. Framework_Laptop_13_Ryzen7040_capsule_signed_allsku_3.07.cap) that contained the actual SPI flash contents. This was my first time dealing with these UEFI capsules, but by using UEFITool, binwalk, and some basic hex comparisons (both on my dumped SPI flash contents and the capsule file from the Framework upgrade zip), I came to the following conclusions:

  1. My SPI flash seemed to have been mostly (if not fully) updated to 3.07. All of the firmware regions matched perfectly with the UEFI capsule file. The differences were only in some non-zeroed padding regions, along with the expected differences in the nvram VSS store region (where BIOS settings and other non-volatile status is saved). I still have no idea which of these two differences was causing it to fail to boot.
  2. The UEFI capsule file had some header stuff, then a couple Microsoft PE-formatted executables, then the actual SPI flash payload, then finally some trailing random data. I didn’t bother digging it into any more than necessary to figure out which region of this file I needed to flash. I primarily used UEFITool to figure out the relative offsets of the regions between my SPI flash dump and the UEFI capsule file.

Once I had my offset and length figured out, I extracted the SPI flash portion of the UEFI capsule as follows:

$ md5sum Framework_Laptop_13_Ryzen7040_capsule_signed_allsku_3.07.cap
ae4043c17d95f36f14f342e6deca3c5e  Framework_Laptop_13_Ryzen7040_capsule_signed_allsku_3.07.cap
$ tail -c +1993293 Framework_Laptop_13_Ryzen7040_capsule_signed_allsku_3.07.cap | head -c 33554432 > framework-3.07-rom.bin 
$ md5sum framework-3.07-rom.bin 
e8241cda9edf65627d84303f9e6f91af  framework-3.07-rom.bin

IMPORTANT NOTE: The offset and md5sums will be different if you’re trying to flash a different version of the BIOS.

Next, I went ahead and flashed away:

$ flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=256 -w framework-3.07-rom.bin --progress 
flashrom v1.6.0-devel (git:v1.5.0-33-gc3b89597) on Linux 6.6.74+rpt-rpi-2712 (aarch64)
flashrom is free software, get the source code at https://flashrom.org

Found Winbond flash chip "W25Q256JW" (32768 kB, SPI) on linux_spi.
Reading old flash chip contents... 
[READ: 100%]...done.
Updating flash chip contents... 
[WRITE: 100%]...Erase/write done from 0 to 1ffffff
Verifying flash... 
[READ: 100%]...VERIFIED.

I then disconnected all my wires, cleaned off the solder flux, booted up my laptop, and breathed a huge sigh of relief when I got my LUKS password prompt. Everything has been working flawlessly for the ~6 hours since doing this.

Also note this will wipe all BIOS settings to default (which I absolutely wanted to do).

After doing all this, I am noticing that fwupdmgr is reporting that it wants to update my UEFI dbx…

$ fwupdmgr get-updates
Devices with no available firmware updates: 
 • WD BLACK SN850X 2000GB
Devices with the latest available firmware version:
 • Fingerprint Sensor
 • System Firmware
Framework Laptop 13 (AMD Ryzen 7040Series)
│
└─UEFI dbx:
  │   Device ID:          362301da643102b9f38477387e2193e57abaa590
  │   Summary:            UEFI revocation database
  │   Current version:    20230501
  │   Minimum Version:    20230501
  │   Vendor:             UEFI:Microsoft
  │   Install Duration:   1 second
  │   GUIDs:              f8ba2887-9411-5c36-9cee-88995bb39731 ← UEFI\CRT_A1117F516A32CEFCBA3F2D1ACE10A87972FD6BBE8FE0D0B996E09E65D802A503&ARCH_X64
  │                       115f7cac-f705-5d34-9a47-37177c3e8514 ← UEFI\CRT_B38FAD316F525F27B27A21B486456C3E4279748BF16893827BF16FE659C0F75E&ARCH_X64
  │   Device Flags:       • Internal device
  │                       • Updatable
  │                       • Supported on remote server
  │                       • Needs a reboot after installation
  │                       • Device is usable for the duration of the update
  │                       • Only version upgrades are allowed
  │                       • Signed Payload
  │ 
  └─Secure Boot dbx Configuration Update:
        New version:      20241101
        Remote ID:        lvfs
        Release ID:       105821
        Summary:          UEFI Secure Boot Forbidden Signature Database
        Variant:          x64
        License:          Proprietary
        Size:             15.1 kB
        Created:          2025-01-17
        Urgency:          High
        Tested by Linux Foundation:
          Tested:         2025-01-20
          Distribution:   fedora 41 (workstation)
          Old version:    20240301
          Version[fwupd]: 2.0.4
        Vendor:           Linux Foundation
        Duration:         1 second
        Release Flags:    • Trusted metadata
                          • Is upgrade
                          • Tested by trusted vendor
        Description:      
        This updates the list of forbidden signatures (the "dbx") to the latest release from Microsoft.
        
        An insecure version of Howyar's SysReturn software was added, due to a security vulnerability that allowed an attacker to bypass UEFI Secure Boot.
        Issues:           529659
                          CVE-2024-7344
        Checksum:         d661d4a0aaca09dfa9e56967ca2467b0575fc07cb704d182fa8c68225452957f

From my very limited experience on UEFI (from going through this recovery process), I learned that is a section in flash with a list of signatures of firmware that should be “blacklisted” for secure-boot purposes. I don’t use secure-boot (shame on me), and considering my shotty experience with the Framework BIOS update process, I’m not touching this with a 10-foot pole. I’ll wait to do anything that writes to this SPI flash until absolutely necessary.

Happy hacking!

10 Likes
4

So, this does show how risky a BIOS upgrade is on the FW13 AMD.

If the BIOS update fails, the motherboard is bricked, unless you use some of the soldering iron based recovery process described above.

Maybe on future motherboards, FW might make it slightly more support friendly.
I.e. make BIOS updates “un-brickable”.

At least we know what “WGGGGGGGGGGGGOGBGBBGGG” means now.
BIOS Code: 00011010 or 0x1A
It means that the BIOS is probably corrupted.

it would have been really interesting to see what the EC console output during this fault.

5

Amazing work and thank you for documenting this.

However, I’m surprised and a little concerned support concluded you’d need to purchase a new motherboard. Excluding any obvious “customer induced damage”, this seems like a software issue. In an ideal world, it’d be RMA’ed for further diagnosis regardless of warranty status (the AMD 7040 series is not that old - began shipping in Oct 2023?).

1 Like
6

Fully agree! I replied to the support email thread yesterday after I got this all working, explained what I did, plus offered to provide them my original/broken flash dump. They responded and simply advised me to “keep on monitoring the device”. Pretty sure I didn’t get past the first-level support there…

If any Framework engineers see this, offer still stands! I’m more than happy to provide the dump and would very much appreciate someone looking into what may have caused this.

3 Likes
7

For what it’s worth, dbx is stored in NVRAM and updated the same way as all other UEFI variables stored in NVRAM. There is no Framework-specific reflashing logic used in updating it. It is not stored in a special flash region.

2 Likes
8

Bravo to one of the more extensive writeups in low level flashing. This is not for the faint of heart.

BTW, those who are in school (or still going through school) this is the reason you pay attention when learning about how hardware knows what it is and where it sits in the world around it. EEPROM is amazing and highly underrated.

For those that are wondering, burning new firmware or new code to a chip came from the fact that PROM chips were literally burned in (written) on a programmer, chip removed, and then inserted into the board it functioned on.

A feature of many motherboards is a dual BIOS setup in case of a bad firmware update. I suspect this may be uncommon in laptops or the implementation is different with custom boards. It would be great if there was a jumper or pair of pins you could short for an amount of time and it reverted back to its factory or backup firmware when it was shipped. This adds complexity and cost though to circuit design I imagine.

4 Likes
9

Sadly, I believe I’ve been bitten by the same update failure. I can’t say for certain as I didn’t note the BIOS versions for the update, but did have a fwupdmgr suggested pair of updates for my AMD 7640U. The system never recovered from the reboot.

I’ve been through the same troubleshooting with support over the last two days.

Thankfully, my board is still under warranty (purchased in June of 2024) and they are sending a replacement.

1 Like
10

Is there a linux command that will backup the bios flash? In a format you can write back using the method you used?
It would seem sensible to back it up before any bios upgrade.

11

Here’s a few other community posts that I believe hit the same issue as me:

  1. From April 2024 - Framework 13 AMD 7040, updating BIOS from 3.03 to 3.05. Looks like this one got bricked. OP described a “graphical glitch” during the update process, but no picture attached.
  2. From June 2024 - Framework 16 (assuming AMD 7040), not sure which BIOS versions were involved. Interestingly, in this case the poster said that after letting the battery die, the system came back up and completed the update. OP provided a picture showing the same “triangle of death” that I encountered.
  3. From January 2025 - Framework 16 AMD 7040, updating BIOS from 3.03 to something (likely 3.05). I’m guessing it got bricked, but can’t say for certain, since the OP only described the system being totally hung and the same graphical glitch (triangle of death) that I hit.

I’d also like to mention that I witnessed this graphical glitch as it happened. It didn’t just suddenly “appear” in a single frame. Instead, it started from middle-ish of the screen (the top of the “glitch”), then proceeded one horizontal line at a time down the screen. It happened relatively slowly - it definitely took at least 20 seconds until the glitch grew to the bottom of the screen. This screams to me like some sort of memory access going off-the-rails…

Really wish I could get the attention of a @framework-qualified-engineer here - I feel like getting the flash dump from the bricked device into the right hands could really help troubleshoot what is going wrong… Last night I was considering hooking my hacky 30 gauge wires back up to the SPI flash so I could flash the bad image on again and confirm I could reproduce the bricked behavior. But then I came to my senses and decided I shouldn’t push my luck by using a 700°F soldering iron multiple times on the same component.

I initially didn’t want to post my bad flash image here in case there was anything sensitive in it, but after considering it more, I’ve decided there is nothing in there I wouldn’t be willing to share publicly.

So, here’s a google drive link to a tarball that contains two files:

  1. fw-bios-bricked.bin: This is the flash dump from my laptop when it was bricked.
  2. fw-bios-3.07-from-capsule.bin: This is the image I wrote (in its entirety) to my SPI flash chip that made my laptop work again. I got this from the 3.07 update capsule file as described in my comment above - you can confirm with the md5sum (sorry I didn’t use sha256 or better) or you can extract it yourself following the instructions I provided.

Hopefully someone sees this and can look into the problem. I have a hunch this has happened to more people, and that reports of it on the forums will ramp up as more warranties begin to expire.

Cheers!

7 Likes
12

That was me. I experienced pretty much the same thing as you.

Thanks for the great write-up. Flashing the memory didn’t occur to me back then, so unfortunately, I went through the FW’s awful warranty pipeline. After a few months, I got a new motherboard. I then thought of dumping the memory on the new board and writing it to the old one, which brought it back to life. I did it with a programmer, and it went smoothly. In the meantime, I got this tool: Glasgow Interface Explorer | Crowd Supply, which I plan to use if I brick it again. Thankfully, updating from 3.05 to 3.07 recently worked as expected. Another tool worth mentioning is https://ofrak.com/, which I successfully used to hack other embedded devices recently and which I’ll use to explore the FW BIOS if I ever have the time.

2 Likes
13

What is really needed is a failsafe/redundant bios like many desktop motherboards have.

5 Likes
14

I cannot thank you enough for this, my fw13 7048U was bricked trying to upgrade from 3.03 to 3.07 last week, yours was the first thread I saw when I came to the forum, I ordered:

and opened a ticket with framework. They asked a few questions (my LED sequence was wggggggggggggrbgbgbggg, my screen wouldn’t turn on at all, didn’t show what yours did) and for me to try all the troubleshooting steps I’d already tried, then said sorry, out of warranty, buy a new $500 mainboard. :frowning:

So amazon shipping was delayed and those items arrived today, I compiled the latest flashrom from git because the latest released version can’t flash this chip. Then I ran this on arch linux:

# flashrom --programmer ch341a_spi --wp-status
flashrom v1.6.0-devel (git:v1.5.0-36-g0cad6dfd) on Linux 6.13.2-zen1-1-zen (x86_64)
flashrom is free software, get the source code at https://flashrom.org

Found Winbond flash chip "W25Q256JW" (32768 kB, SPI) on ch341a_spi.
Protection range: start=0x00000000 length=0x00000000 (none)
Protection mode: disabled
# flashrom --programmer ch341a_spi -r bob1.bin --progress
flashrom v1.6.0-devel (git:v1.5.0-36-g0cad6dfd) on Linux 6.13.2-zen1-1-zen (x86_64)
flashrom is free software, get the source code at https://flashrom.org

Found Winbond flash chip "W25Q256JW" (32768 kB, SPI) on ch341a_spi.
Reading flash... 
[READ: 100%]...done.
# flashrom --programmer ch341a_spi -r bob2.bin --progress
flashrom v1.6.0-devel (git:v1.5.0-36-g0cad6dfd) on Linux 6.13.2-zen1-1-zen (x86_64)
flashrom is free software, get the source code at https://flashrom.org

Found Winbond flash chip "W25Q256JW" (32768 kB, SPI) on ch341a_spi.
Reading flash... 
[READ: 100%]...done.
# sha1sum bob*
2f2cee73a97863862effcae28deb8ff2474c0143  bob1.bin
81f34a15a89beb737759080eba7cfa94295a7686  bob2.bin

But those two reads were different, so I kept reading to bob3.bin and bob4.bin until finally I got 2 identical reads. This is when I rigged up something to hold it down because flashing and reading take forever and my fingers hurt.

pic-under
pic-under1440×1920 429 KB

pic-over
pic-over1920×1440 370 KB

That’s a pair of helping hands holding the pogo plug and a stack of heavy ceramic coasters on top. Don’t bump the table because the coasters could destroy your machine. :smiley: (yes I tried other things before coasters but everything was either too light or too slippery)

Then I wrote the new rom, but with verification turned off, because it’s slow and I like doing it manually:

# flashrom --programmer ch341a_spi -nNw orig/fw-bios-3.07-from-capsule.bin --progress
flashrom v1.6.0-devel (git:v1.5.0-36-g0cad6dfd) on Linux 6.13.2-zen1-1-zen (x86_64)
flashrom is free software, get the source code at https://flashrom.org

Found Winbond flash chip "W25Q256JW" (32768 kB, SPI) on ch341a_spi.
Reading old flash chip contents... 
[READ: 100%]...done.
Updating flash chip contents... 
Fixing total value of stage 2 progress on the fly.
[READ:  0%][ERASE: 100%][WRITE: 100%]...Fixing total value of stage 1 progress on the fly.Fixing total value of stage 1 progress on the fly.Fixing total value of stage 1 progress on the fly.Fixing total value of stage 1 progress on the fly.Fixing total value of stage 1 progress on the fly.Fixing total value of stage 1 progress on the fly.Fixing total value of stage 1 progress on the fly.Fixing total value of stage 1 progress on the fly.Fixing total value of stage 1 progress on the fly.Fixing total value of stage 1 progress on the fly.Fixing total value of stage 1 progress on the fly.Fixing total value of stage 1 progress on the fly.Fixing total value of stage 1 progress on the fly.Fixing total value of stage 1 progress on the fly.Erase/write done from 0 to 1ffffff

Then I read the rom back, and verified the sha1sum matched what I just wrote.

Then I put the ram+nvme back in and plugged in the power and it works.

I can upload the bad firmware for whoever wants it.

10 Likes