Script to corrupt openpilot routes

[vishalkrishnads]

Malformer

This is a python CLI tool to corrupt good openpilot routes in a format that can be easily uploaded to the API. Fixes #39. It uses cereal's log.capnp to parse existing logs and create new ones. An obvious disadvantage is having to setup and maintain a python environment within this project, but there doesn't seem to be any JS solutions to effectively handle capnp messages. More on that below.

Usage

Make sure to run pip install -r malformer/requirements.txt first. There are basically three arguments required for running malformer:

• The account token, which can be passed in using the -a or --account flags
• The device's dongle id, which can be passed in using the -d or --device flags
• The route ID, which can be passed in using the -r or --route flags

An example usage with the default values would be

~/new-connect$ python malformer -d '1d3dc3e03047b0c7' -r '000000dd--455f14369d' -a 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3NDg1ODI0NjUsIm5iZiI6MTcxNzA0NjQ2NSwiaWF0IjoxNzE3MDQ2NDY1LCJpZGVudGl0eSI6IjBkZWNkZGNmZGYyNDFhNjAifQ.g3khyJgOkNvZny6Vh579cuQj1HLLGSDeauZbfZri9jw'

What works?

Basically, any field that you want removed can be removed from the logs. The field name should be specified in the CAPNP_MAP dict. Stuff like extending the segment length over 60s is still pending. Once specified, you can use the UI to select which all to remove, no need to edit the file for each route.

Why not JavaScript?

Currently, there seem to be 3 choices in JS for processing capnp files/messages:

1. capnp-ts - the most robust one, but misses out on the ability to write to capnp files
2. node-capnp - the implementation is slow
3. capnp-stream - doesn't seem to offer methods to effectively alter individual objects.

[github-actions]

deployed preview: https://43.connect-d5y.pages.dev

Welcome to new-connect! Make sure to:

• read the contributing guidelines
• mark your PR as a draft until it's ready to review
• post the preview on Discord; feedback from users will speedup the PR review

Mobile

Desktop

[adeebshihadeh]

This PR and its description are massive. Can you clean this up first? This should be like 100 lines, 200 max.

[vishalkrishnads]

I've merged all the code into a single file as you directed and brought the code down to ~200 lines. Please check and inform me what further changes you need. I couldn't find anything else to remove from the description either.

[knownotunknown]

Hey @vishalkrishnads, you have a check falling, but otherwise maybe you should also set this PR to open rather than draft? And probably ping to follow up again.

[vishalkrishnads]

Oh cool @knownotunknown I'll set it to open, but the check failing doesn't seem to be related to the build failing or anything. I've been having issues with the cloudflare deployment in this repo for quite a while now. As you can see, the build succeeds but the preview fails, not sure what's going on still.

[incognitojam]

Thanks for the PR but I decided to approach this in a different way