comma 3X: no panda due to invalid CR3 config

[adeebshihadeh]

So you installed a development branch and your panda won't come up? Fortunately, there's a simple recovery procedure.

Notes about the issue: commaai/panda@5ee262f.

Recovery procedure

It's recommended to run this on a Linux machine, but it can be made to work on other machines too.

Prep:

• 2 USB-C cables
• install master-ci on your comma 3X: installer.comma.ai/commaai/master-ci
• setup your PC
  • install fastboot
  • git clone https://github.com/commaai/agnos-builder
  • download this, unzip it, and move prog_firehose_ddr.elf to the root of agnos-builder/
• understand that OBD-C is port #1 and aux is port #2

Steps:

1. power device through OBD-C port for 1 minute
2. remove OBD-C power and plug into the aux port. (it's expected that the device's screen will be blank until after step 7 is run)
3. cd into agnos-builder on your PC
4. run tools/edl --loader=prog_firehose_ddr.elf getstorageinfo
5. wait 10 minutes (the probability this works goes up the longer you wait)
6. plug power into the OBD-C port
7. run tools/edl reset && fastboot continue
8. wait for the openpilot UI to show up with "no panda", then wait another 2 minutes
9. fully unplug device and let the caps drain for 5 minutes
10. plug it in and your panda should be back up

⚠️ This process isn't perfectly repeatable, so it might take repeating a few times before it works.

[connoralydon]

on Apple Silicon Mac. I had to use Rosetta 2 and prefix the toold/edl command with arch -x86_64 as well as pip install libusb.

[connoralydon]

this worked, if you need some help or troubleshooting I'm @lydonfyi on X

[itsTreble]

confirm this fixed no "no panda" issue caused by me flashing sunnypilot dev-c3 on my new c3x right out of the box.

I had no luck doing this on a windows machine, even tried WSL. Alternative would be to create a linux VM or just make a bootable linux USB and run the recovery from there. I chose the latter and got my c3x back to "vehicle online"!

[bravochar]

For anyone else trying to do this with WSL, I appear to be having some luck with usbipd.exe. I had an error when initially connecting to the AUX port to Windows, but I unplugged and flipped the USB-C connector (I know that shouldn't make a difference, but sometimes it does if cables/connectors are bad). After running usbipd.exe bind --busid <bus-id> in an Adminstrator Powershell and usbipd.exe attach --wsl --busid 10-4 in a regular Powershell, I was able to see the device in my Ubuntu WSL shell with lsusb. I had to run the edl commands as sudo, though, or I just got nonstop "Couldn't get device configuration" errors. Nevermind, I had to do it on MacOS with the above instructions.

Provided that my device is recovered with this procedure, will I be able to install a custom branch again without invalidating the CR3 config? It's not clear from anything above what caused the problem, and I'd hate to go through the trouble of fixing it only to cause it again. Should I rebase my custom branch off of master-ci instead, or are there any other steps I should take?

[itsTreble]

Provided that my device is recovered with this procedure, will I be able to install a custom branch again without invalidating the CR3 config? It's not clear from anything above what caused the problem, and I'd hate to go through the trouble of fixing it only to cause it again. Should I rebase my custom branch off of master-ci instead, or are there any other steps I should take?

I was able to get panda back with the recovery steps, reset it then load dev-c3 sunnypilot without any issue.

adeeb has a summary what happened here: commaai/panda@5ee262f

[AskAlice]

This is happening on my comma 3, not just 3x ig

[ricardohalfeld]

This also happened to my Comma 3X. The only noteworthy event between it working fine and showing up with "No Panda" is that I unplugged it without powering it down first.

Anyway, I followed the steps and was beginning to wonder how many retries were too many, then it worked. I did the entire thing 4 or 5 times, for reference.

I'd modify the commands to change some of the permissions right before running them. I read somewhere that all activates need +rw, and they change back quickly somehow. Like this:

chmod +rw ~/agnos-builder/tools/edl_repo/venv/bin/?ctivate* && tools/edl --loader=prog_firehose_ddr.elf getstorageinfo
chmod +rw ~/agnos-builder/tools/edl_repo/venv/bin/?ctivate* && tools/edl reset && fastboot continue