Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a small tool to iterate a car file and detect missing metadata #158

Open
linuskendall opened this issue Sep 12, 2024 · 4 comments
Open
Assignees
Labels

Comments

@linuskendall
Copy link
Contributor

We need a tool that iterates a car file and identifies which transactions are missing metadata.

@linuskendall
Copy link
Contributor Author

Replaces #154

@AzothZephyr
Copy link

gm, so im interested in taking this and #154 on early next week if there's limited capacity available. im coming in mildly half cocked but from what @notwedtm has expressed some early epochs contain transactions which are missing the metadata field.

what are your thoughts around a --verify flag which takes a directory then iterates over each .car file within that directory to execute verify logic on each tx within it to close this issue? that same verification logic could then be applied at the end of a car files generation to verify the underlying data as close to source as possible then potentially expanded out to cover other cases elsewhere in the dag if need be, which should close #154 as well.

im still going through the codebase but i think this is a good approach as it gives the ability for operators who are already using / hosting car files the ability verify their store without having to blindly pull after a fix is produced and insures we don't see this issue without some form of visibility moving forward.

will move forward next week unless @gagliardetto already has a solution cooking

az

@Lusitaniae
Copy link
Contributor

Lusitaniae commented Sep 19, 2024

checking a single file could be best, in case people don't have all CAR files locally and it would be easier to parallelized using xargs/parallel on the invocation

@AzothZephyr
Copy link

very good point, i will add a subcommand for --file / -f and --dir / -d. doesn't add much work wise but covers for both cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants