Flashed modified firmware, laptop not booting anymore

Hello everyone,
I modified the embedded controller (EC) firmware of my 11th gen Framework following this blog post, I compiled the firmware and then flashed it successfully using ECTool.
Rebooted the computer and it doesn’t start, the power button turns on for a second and then turns off.

Before I flashed the (probably broken) firmware I modified, I made a backup of the original firmware, so if I could flash it back on EC i could probably fix the laptop.

In the blog post I linked before it says that on the board there is a JECDB connector, but I cannot find any info about it, only a pinout on the Framework EC firmware repo.

Thank you in advance for the help.
Hope this won’t become the excuse to buy a 12th gen board.

Ugh, as the author of that blog post I am sorry!

I think I know how this happened. As a quick check… can you run arm-none-eabi-gcc --version?

I suspect it is this:

Yep, the gcc version was higher than 11, in fact I used GCC 12.2.0
I saw that you tested it was tested only with GCC 5, but I proceded anyway.
I should have checked better :smiling_face_with_tear:

Anyway, the blog post is very cool, and I understand that following your steps (maybe not too closely) was a risk and I’m responsible for it.

Maybe just to be sure this dosen’t happen to anyone else you could put a disclaimer on the blog post.

Done! It’s publishing now. I should have updated it the moment I nearly bricked my own machine¹.

¹ Unfortunately, I was saved by a fluke. I’ve been working on getting the 2-image firmware system going again, and while the image I had flashed couldn’t boot the AP it could boot the unmodified second copy of the firmware. Alas.

2 Likes

If you’ve got access to the build tree for your EC image, can you make sure you archive build/hx20? I’d love to grab a copy of it. It could aid in recovery!

I’ll give it to you in a moment, I need to boot into the nvme drive that was in the laptop.

By reading the pinout of the JECDB header on the board, it seems that it has connectors for UART and JTAG, so with a JTAG programmer is it possible to write to the flash memory? or it’s much more complicated than that?

I’m not quite sure about that, actually. I’ve only used it for UART, and only then by meticulously pressing leads against it. :slightly_smiling_face:

Ok, I uploaded my the files http://cerea.dev/mistakes.tar.xz

Forgot to say it before thank you very for your help and fast response too!!!

2 Likes

Hey, as you’re one of the two other people I know doing EC work I’d be remiss in not helping you out.

I’ll put together a plan of action over the next couple of days, but I’m pretty sure we can get you back to right.

4 Likes

Okay, I’ve tested this out on my own machine and it works.

If you can get to the UART, run these commands. They’re specific to the exact firmware image you flashed, so they won’t get anyone else out of the same bind. Remember that it’s 3V3, not 5V or 1V8! You will probably want to copy/paste them in 6-line chunks so as to not overwhelm the console driver’s input buffer.

Theory of Operation
  • The issue is in the code generation for two functions that trigger when the CPU starts up (interrupt handlers for ESPI.)
  • The EC doesn’t have XIP or mapped flash. Therefore, all code is in RAM.

With access to (1) the UART console, (2) the memory map for the specific firmware that you built, and (3) some tenacity… we could author replacements somewhere in unused RAM and then replace the prologue of those functions with a jump there instead.

The EC console has some primitive memory writing commands.

Recovery Commands
rw 0x101320 0x41f0e92d
rw 0x101324 0x4f144b13
rw 0x101328 0x2144f8d3
rw 0x10132c 0x5148f8d3
rw 0x101330 0x8048f8df
rw 0x101334 0x5140f8c3
rw 0x101338 0xf4a5fa95
rw 0x10133c 0xf484fab4
rw 0x101340 0x2c20260c
rw 0x101344 0xe8bdd101
rw 0x101348 0x08a281f0
rw 0x10134c 0x0303f004
rw 0x101350 0x3302fb06
rw 0x101354 0x5dd84621
rw 0x101358 0x3024f858
rw 0x10135c 0x0001f000
rw 0x101360 0x23014798
rw 0x101364 0xea2540a3
rw 0x101368 0xfa950503
rw 0x10136c 0xfab4f4a5
rw 0x101370 0xe7e6f484
rw 0x101374 0x4000e000
rw 0x101378 0x400f9c08
rw 0x10137c 0x000fc528
rw 0x101380 0x41f0e92d
rw 0x101384 0x4f144b13
rw 0x101388 0x2158f8d3
rw 0x10138c 0x515cf8d3
rw 0x101390 0x8048f8df
rw 0x101394 0x5154f8c3
rw 0x101398 0xf4a5fa95
rw 0x10139c 0xf484fab4
rw 0x1013a0 0x2c20260c
rw 0x1013a4 0xe8bdd101
rw 0x1013a8 0x08a281f0
rw 0x1013ac 0x0303f004
rw 0x1013b0 0x3302fb06
rw 0x1013b4 0x5dd84621
rw 0x1013b8 0x3024f858
rw 0x1013bc 0x0001f000
rw 0x1013c0 0x23014798
rw 0x1013c4 0xea2540a3
rw 0x1013c8 0xfa950503
rw 0x1013cc 0xfab4f4a5
rw 0x1013d0 0xe7e6f484
rw 0x1013d4 0x4000e000
rw 0x1013d8 0x400f9c5c
rw 0x1013dc 0x000fc598
rw 0x1013e0 0xb5014670
rw 0x1013e4 0xf94ef7f6
rw 0x1013e8 0xff9af7ff
rw 0x1013ec 0x4001e8bd
rw 0x1013f0 0xb96af7f6
rw 0x1013f4 0xb5014670
rw 0x1013f8 0xf944f7f6
rw 0x1013fc 0xffc0f7ff
rw 0x101400 0x4001e8bd
rw 0x101404 0xb960f7f6
rw 0xe107c 0x1013e1
rw 0xe1080 0x1013f5

What we’re doing is this:

  1. Writing a new version of the miscompiled code into slack space in EC RAM (unused after first boot); this makes up the bulk of the input
  2. Overwriting the interrupt vector table entries to point at the new code; this is the last two lines

This is all generated from the code in this repository, which you can audit/check out/compile as you wish. Running make after providing the offsets from your firmware map will produce a new espi_script.txt which should have the same contents as mine, assuming a similar compiler version. :slightly_smiling_face:

After that’s done, click the power button. Your machine should boot up. Immediately go restore your original firmware image!

The changes we’re making here only apply to the code in memory. If your EC reboots, they’ll be lost and you’ll be back where you were. It’s non-destructive, and it should get you booting.

2 Likes

I’m waiting for the UART adaptor to arrive, when it’s here I’ll tell you how it went

4 Likes

I tried connecting with the UART adapter, but I didn’t work, (1) The software on my pc is not setup correctly (I’ll do some testing with an Arduino to see if the UART adapter works) (2) I don’t know the baud rate of the arm chip do you know it by any chance.

You’ll want 115200 8N1 with no flow control! Good luck!

1 Like

It worked!!!
Here is what I have done, Yesterday I soldered two wires to JECDB port to connect to UART, but I didn’t get any output because the USB cable was broken (bruh). Then while moving everything I ripped off the pad for TX (bruh x2)
Today, using continuity mode on a multimeter, I found out that the TX pin was also on the JSWDB port (the one with 12 pins) on pin 4 starting from the left.
So I soldered the wires run your commands and then boom the computer booted.



13 Likes

HOT DAMN! I can’t believe it worked!

Congratulations :slight_smile:

7 Likes

Pure wizardry to those of us without your skills!

4 Likes

Unfortunately, this happened to me as well :man_facepalming:… this time it was with gcc 12 but it was just a vanilla embedded controller firmware build with no modifications. The problem im having is that the soldering iron I own is far too large for the tiny pads and will probably burn them off, I believe the offsets I generated are correct but if anyone could verify the smap its here: anotherpersonwhomademistakes/ec.RO.smap at build · ladysman-217/anotherpersonwhomademistakes · GitHub

I really hope I didnt trash a few hundred dollars worth of mainboard…

Edit: Guess I should contact @DHowett as I just realized he made the espi recovery script…

You almost had an excuse to buy a new soldering iron. So close!
I’d use it as an excuse to get a pine64 pinecil

I really love my Pinecil!

@Autoboticon, I reached out over PM to give you a couple additional details; your map looks right, and I did give it a quick run through framework-espi-recovery to generate this:

ESPI UART recovery for Autoboticon
rw 0x101560 0x41f0e92d
rw 0x101564 0x4f144b13
rw 0x101568 0x4b14681b
rw 0x10156c 0x8050f8df
rw 0x101570 0xf843681c
rw 0x101574 0xfa944c08
rw 0x101578 0xfab5f5a4
rw 0x10157c 0x260cf585
rw 0x101580 0xe8bdb90c
rw 0x101584 0xf00581f0
rw 0x101588 0x08ab0203
rw 0x10158c 0x2303fb06
rw 0x101590 0x5dd84629
rw 0x101594 0x3025f858
rw 0x101598 0x0001f000
rw 0x10159c 0x23014798
rw 0x1015a0 0xf505fa03
rw 0x1015a4 0x0405ea24
rw 0x1015a8 0xf5a4fa94
rw 0x1015ac 0xf585fab5
rw 0x1015b0 0xbf00e7e6
rw 0x1015b4 0x4000e144
rw 0x1015b8 0x400f9c08
rw 0x1015bc 0x4000e148
rw 0x1015c0 0x000fc78c
rw 0x1015c4 0x41f0e92d
rw 0x1015c8 0x4f144b13
rw 0x1015cc 0x4b14681b
rw 0x1015d0 0x8050f8df
rw 0x1015d4 0xf843681c
rw 0x1015d8 0xfa944c08
rw 0x1015dc 0xfab5f5a4
rw 0x1015e0 0x260cf585
rw 0x1015e4 0xe8bdb90c
rw 0x1015e8 0xf00581f0
rw 0x1015ec 0x08ab0203
rw 0x1015f0 0x2303fb06
rw 0x1015f4 0x5dd84629
rw 0x1015f8 0x3025f858
rw 0x1015fc 0x0001f000
rw 0x101600 0x23014798
rw 0x101604 0xf505fa03
rw 0x101608 0x0405ea24
rw 0x10160c 0xf5a4fa94
rw 0x101610 0xf585fab5
rw 0x101614 0xbf00e7e6
rw 0x101618 0x4000e158
rw 0x10161c 0x400f9c5c
rw 0x101620 0x4000e15c
rw 0x101624 0x000fc7fc
rw 0x101628 0xb5014670
rw 0x10162c 0xf938f7f6
rw 0x101630 0xff96f7ff
rw 0x101634 0x4001e8bd
rw 0x101638 0xb954f7f6
rw 0x10163c 0xb5014670
rw 0x101640 0xf92ef7f6
rw 0x101644 0xffbef7ff
rw 0x101648 0x4001e8bd
rw 0x10164c 0xb94af7f6
rw 0xe107c 0x101629
rw 0xe1080 0x10163d

Not all is lost. Those little pads can be finicky, but… the first few times I was messing around I did so by holding jumper wire down against the ones I needed. It’s tedious, and having an extra set of hands helps…

2 Likes