feat: `atrium-repo` #272

DrChat · 2024-12-23T20:14:30Z

This builds on str4d's work here: #168.

Big notes:

Complete merkle search tree implementation (but still lacking fuzz tests)
Added a new pair of traits for block storage: AsyncBlockStoreRead and AsyncBlockStoreWrite.
- In-memory storage backend (primarily intended for testing)
- CAR storage (supporting both reading and writing)
- Users can implement other storage sources, like Azure-backed block storage.

Merkle Search Tree
- Parse Node
  - Read Node from bytes
  - Verify depth and sort order
  - Limit the number of TreeEntrys per Node to a statistically unlikely maximum length.
  - Consider limiting the overall depth of the repo, or other parameters, to prevent more sophisticated key mining attacks.
- Locate key within node
- Locate keys within node with a given prefix
- Add entry
- Edit entry
- Delete entry
Repository
- Load from a CAR file
- Load from a firehose record
- Enumerate all keys
- Fetch all records in a given collection
- Get a specific record
Storage
- CAR files
  - Reading CAR files
  - Verify completeness of the repository structure.
  - Robustness to both duplication and de-duplication of blocks.
  - Ignore any unnecessary or unlinked blocks.
- Commits from firehose records

…ost position in parent

…t firehose records

DrChat · 2025-01-01T20:11:19Z

atrium-repo/src/mst.rs

+        loop {
+            let node = Node::read_from(&mut bs, node_cid).await?;
+            if !seen.insert(node_cid) {
+                // This CID was already seen. There is a cycle in the graph.


Is it even possible to serialize a graph with a cycle?
The two nodes would point at each other, and by definition it should be impossible to compute their CIDs if there is a cycle.

Why: This function is a footgun that will likely be misused. Deleting a block is not typically safe because it could have multiple incoming references (and thus deletion converts those into dangling references). The only way to safely delete a block would be to wholistically enumerate an archive file and collect a list of all referenced blocks.

DrChat · 2025-01-18T22:24:33Z

I think the PR is in a pretty good state for a MVP.
My time is limited so I won't be able to quickly iterate on the functionality for creating new commits for a repository. Still planning on it, but the work here can be pushed up in the meantime :)

erlend-sh · 2025-01-19T12:29:03Z

Pinging @str4d since this builds on their prior work.

sugyan

I have written several comments on areas unrelated to the internal logic. There are also a couple of points from clippy that I hope you can correct as well.

sugyan · 2025-01-19T13:50:01Z

.gitignore

@@ -12,5 +12,8 @@ target/
 # These are backup files generated by rustfmt
 **/*.rs.bk

+# VSCode settings
+/.vscode


I do not want to include anything related to a specific editor.

I added this line because VSCode annoyingly creates this folder automatically (likely due to the Copilot extension being broken and populating a settings.json).
If it isn't present, it's almost certain that I or someone else may accidentally commit this folder.

If you don't want to have a line that explicitly mentions VSCode, another alternative you could consider is changing this .gitignore file to an allowlist.
Here's an example from one of my other projects.

Each individual's settings related to the editor should have their own global ignore settings, and I don't want to include changes on the repository side for that.

According to man gitignore:

Patterns which a user wants Git to ignore in all situations (e.g., backup or temporary files generated by the user’s editor of choice) generally go into a file specified by core.excludesFile in the user’s ~/.gitconfig. Its default value is $XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is either not set or empty, $HOME/.config/git/ignore is used instead.

I believe that editor related ignore settings should be written there.

sugyan · 2025-01-19T13:51:19Z

Cargo.toml

-    "examples/concurrent",
-    "examples/firehose",
-    "examples/video",
+    # "examples/concurrent",


Is there a reason why these are no longer excluded?

Ah - I had forgotten about this.
I changed this because the examples were written against the last released version of atrium rather than the version present in the repository.
I needed to update the firehose example to incorporate the changes I have made here, and chose to change its dependencies to point at the crates in this repository.

The decision to have examples demonstrate the last public release seems to be an unusual one. I haven't ever seen this be done for any other Rust library.
Any thoughts about changing this?

Indeed. The current policy of using the latest release version does not allow us to use newly added packages.

We've discussed this before and made it the way it is now,
#100 (comment)

But maybe we should reconsider.

sugyan · 2025-01-19T13:52:43Z

atrium-repo/Cargo.toml

+[package]
+name = "atrium-repo"
+version = "0.0.0"
+authors = ["Jack Grigg <[email protected]>"]


I think it would be good to include both of your names.

sugyan · 2025-01-19T13:54:30Z

Cargo.toml

+
+# async in traits
+# Can be removed once MSRV is at least 1.75.0.
+async-trait = "0.1.68"


This has already been removed by raising the MSRV.

sugyan · 2025-01-19T13:58:09Z

atrium-repo/README.md

+[![](https://img.shields.io/crates/v/atrium-repo)](https://crates.io/crates/atrium-repo)
+[![](https://img.shields.io/docsrs/atrium-repo)](https://docs.rs/atrium-repo)
+[![](https://img.shields.io/crates/l/atrium-repo)](https://github.com/sugyan/atrium/blob/main/LICENSE)
+[![Rust](https://github.com/sugyan/atrium/actions/workflows/api.yml/badge.svg?branch=main)](https://github.com/sugyan/atrium/actions/workflows/api.yml)


Since this package also contains tests, you should also add workflows/repo.yml and this should be a link to it.

DrChat marked this pull request as draft December 23, 2024 20:14

DrChat force-pushed the repo branch 5 times, most recently from 4b54d5e to 3b84377 Compare December 26, 2024 20:54

str4d and others added 10 commits December 26, 2024 15:06

chore: Create empty atrium-repo library crate

7095370

repo: Logic for parsing and traversing MST nodes

4f12317

repo: Indexed reader for CAR files

7a1b4c9

repo: APIs to fetch keys, collections, and records from a repository

30813f9

Significant refactor; implement basic MST update logic

fb910a2

Full MST implementation

510f9fc

More refactoring :)

44a6ea2

Use traverse for split_subtree

2c06a6c

Fix a few lingering bugs in node deletion

602a4dc

Update firehose example

f8e988e

DrChat force-pushed the repo branch from 3b84377 to f8e988e Compare December 26, 2024 21:08

DrChat added 13 commits December 26, 2024 18:37

More validation in mst dual-layer deletion test

43e2cb6

Tree::get_path

37db11f

Only verify SHA2_256 CAR blocks

4ee5c3c

Documentation pass

98dcc93

MST: Handle edge case where we need to insert new tree entry at leftm…

0958688

…ost position in parent

Fix repo tests

a383769

More docs

d008bc0

Simplify Tree::keys

cbb301b

Return keys from Tree::keys in lexicographic order

49fb8df

Create a high-level Commit helper struct

256365f

Add new Tree::entries function to return key/value tuples

50b5cd6

Add documentation clarifying that tree iteration does not work agains…

6655a83

…t firehose records

Whoops :)

24b2bf1

DrChat added 4 commits December 30, 2024 15:43

Expose hashing algorithm to AsyncBlockStoreWrite::write_block

e70e2ff

Forgot to commit this import :(

e16a4bf

Gracefully handle multi-writes to CarStore

f0f8803

Rename Repository::new to open

40f2eca

DrChat force-pushed the repo branch from f2ded02 to f736ccc Compare January 1, 2025 19:14

Outline algos::compute_depth

b230066

DrChat force-pushed the repo branch from f736ccc to b230066 Compare January 1, 2025 19:20

DrChat added 2 commits January 1, 2025 13:48

Correct the docs for Node::find_ge

3507531

Panic if we attempt to serialize a Node with two adjacent tree entries

766528e

DrChat commented Jan 1, 2025

View reviewed changes

DrChat added 6 commits January 3, 2025 16:08

Misc. improvements

86c6d90

More Tree documentation

f27ad5d

Fix a bug when writing out a new CAR block

4ee9bb9

API updates; Implement CarStore::create

6281c10

Update firehose example to remove CID compat

c045e02

DrChat force-pushed the repo branch from 70e1cfc to c045e02 Compare January 12, 2025 23:00

DrChat added 2 commits January 18, 2025 15:54

Move all algorithms to algos module

e52792b

Fix warnings

60adc4a

DrChat force-pushed the repo branch from 6db3530 to 60adc4a Compare January 18, 2025 21:58

DrChat marked this pull request as ready for review January 18, 2025 22:24

sugyan self-requested a review January 19, 2025 13:48

sugyan requested changes Jan 19, 2025

View reviewed changes

clippy + fmt

a76b0e7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: `atrium-repo` #272

feat: `atrium-repo` #272

DrChat commented Dec 23, 2024 •

edited

Loading

DrChat Jan 1, 2025

DrChat commented Jan 18, 2025

erlend-sh commented Jan 19, 2025 •

edited

Loading

sugyan left a comment

sugyan Jan 19, 2025

DrChat Jan 19, 2025

sugyan Jan 20, 2025

sugyan Jan 19, 2025

DrChat Jan 19, 2025

sugyan Jan 20, 2025

sugyan Jan 19, 2025

sugyan Jan 19, 2025

sugyan Jan 19, 2025

feat: atrium-repo #272

Are you sure you want to change the base?

feat: atrium-repo #272

Conversation

DrChat commented Dec 23, 2024 • edited Loading

Choose a reason for hiding this comment

DrChat commented Jan 18, 2025

erlend-sh commented Jan 19, 2025 • edited Loading

sugyan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feat: `atrium-repo` #272

feat: `atrium-repo` #272

DrChat commented Dec 23, 2024 •

edited

Loading

erlend-sh commented Jan 19, 2025 •

edited

Loading