Large object storage (#125)

vsql now supports storing objects (rows, tables, etc) larger than a page (which previously was a serious limitation). This happens entirely transparently on the B-tree implementation where larger objects are split (and reconstructed) to and from new `B` (blob) and `F` (fragment) objects. This is a breaking change to the file format. Some other notable improvements: - Documentation now has Excalidraw diagrams. The "File Format" page has been totally overhauled and is a lot easier to understand and much nicer to look at: https://vsql.readthedocs.io/en/latest/file-format.html - Added "Limitations" page. Among which, a vsql file - in theory - can hold up to 128PB (wholly untested, of course). - Page splitting now has a heavy bias towards filling the left page as much as possible (rather than even splitting). We prefer the left page to be a full as possible because the keys are sequential, so in the case of lots of sequential inserts it can pack the data more tightly and causes less page splits as new data is inserted. - Added another B-tree test for testing 100k sequential inserts/deletes and improved the existing random insert/remove test matrix from just 48 byte objects to also include 148 byte objects (one object per page) and 348 byte objects (always using blob storage). Fixes #43
elliotchance · Aug 4, 2022 · 7ee3bfd · 7ee3bfd
1 parent 2c4c6f1
commit 7ee3bfd
Show file tree

Hide file tree

Showing 27 changed files with 11,326 additions and 507 deletions.
diff --git a/docs/file-format.rst b/docs/file-format.rst
diff --git a/docs/img/file-format-blob-1.excalidraw b/docs/img/file-format-blob-1.excalidraw
diff --git a/docs/img/file-format-blob-1.png b/docs/img/file-format-blob-1.png
diff --git a/docs/img/file-format-blob-2.excalidraw b/docs/img/file-format-blob-2.excalidraw
diff --git a/docs/img/file-format-blob-2.png b/docs/img/file-format-blob-2.png
diff --git a/docs/img/file-format-blob-3.excalidraw b/docs/img/file-format-blob-3.excalidraw
diff --git a/docs/img/file-format-blob-3.png b/docs/img/file-format-blob-3.png
diff --git a/docs/img/file-format-layout.excalidraw b/docs/img/file-format-layout.excalidraw
diff --git a/docs/img/file-format-layout.png b/docs/img/file-format-layout.png
diff --git a/docs/img/file-format-leaf-page.excalidraw b/docs/img/file-format-leaf-page.excalidraw
diff --git a/docs/img/file-format-leaf-page.png b/docs/img/file-format-leaf-page.png
diff --git a/docs/img/file-format-non-leaf-page.excalidraw b/docs/img/file-format-non-leaf-page.excalidraw
diff --git a/docs/img/file-format-non-leaf-page.png b/docs/img/file-format-non-leaf-page.png
diff --git a/docs/img/file-format-non-leaf-reference.excalidraw b/docs/img/file-format-non-leaf-reference.excalidraw
diff --git a/docs/img/file-format-non-leaf-reference.png b/docs/img/file-format-non-leaf-reference.png
diff --git a/docs/img/file-format-object.excalidraw b/docs/img/file-format-object.excalidraw
diff --git a/docs/img/file-format-object.png b/docs/img/file-format-object.png
diff --git a/docs/img/file-format-page.excalidraw b/docs/img/file-format-page.excalidraw
diff --git a/docs/img/file-format-page.png b/docs/img/file-format-page.png
diff --git a/docs/internals.rst b/docs/internals.rst
@@ -5,4 +5,5 @@ Internals
    :maxdepth: 1
 
    file-format.rst
+   limitations.rst
    mvcc.rst
diff --git a/docs/limitations.rst b/docs/limitations.rst
@@ -0,0 +1,28 @@
+Limitations
+===========
+
+These have not been tested, but are based on fundamental design choices:
+
+.. list-table::
+  :header-rows: 1
+
+  * - Description
+    - Limit
+
+  * - The maximum page size (when creating a new file)
+    - 65536 bytes (64 KB)
+
+  * - The maximum number of pages in a file
+    - 2147483647 (~2.1 billion)
+
+  * - The maximum file size (using the default page size)
+    - 8 PB
+
+  * - The maximum file size (using the maximum page size)
+    - 128 PB
+
+  * - The maximum object size (using the default page size)
+    - 8 TB
+
+  * - The maximum object size (using the maximum page size)
+    - 128 TB