sqld: deduplicate data stored in wallog

**!!! early draft, full of debug prints, barely works !!!**

This draft contains experiments around deduplicating our own `wallog` format
with libSQL WAL. The potential win here is reducing write and space
amplification from 2x to around 1.08x.

The main idea is as follows: `wallog` is only used to store frame metadata, and
frame data is only stored either in the main database file, or in WAL. That's
very simple to implement in a single-node system, but it gets complicated with
replicas, because a replica is allowed to ask the primary for any arbitrary
wallog frame.

The rough idea for dealing with replicas is to:
1. Make sure that we control checkpoints. autocheckpoint is off, and we only
issue a checkpoint operation on the primary ourselves, explicitly, and
periodically.
2. All streaming of frames to replicas must finish before we issue a checkpoint
operation.
3. We only checkpoint in TRUNCATE mode, i.e. a write lock is taken and the
whole WAL log is rewritten to the main db file. That simplifies lots of edge
(sic!) cases.
4. Once we checkpoint, we drop the previous `wallog`, and instead only store
the following information. Let's assume that the main db file has N pages.
Pages 1..N are now available as frames X..X+N in the `wallog`, and X is the
oldest frame a replica should ever ask for -> anything before X is out-of-date
anyway. If any replica asks for an earlier page, it gets an error message
saying "please drop whatever you're doing and start asking for frames X or
greater instead.

Author: psarna

sqld/src/replication/frame.rs

@@ -34,6 +34,66 @@ pub struct Frame {
     data: Bytes,
 }
 
+#[repr(transparent)]
+#[derive(Clone, Copy, Debug, Zeroable, Pod)]
+// NOTICE: frame number 0 indicates that the frame is in the main db file.
+// Any other number indicates that it's in the WAL file.
+// We do not use an enum here in order to make this struct transparently
+// serializable for C code and on-disk representation.
+pub struct FrameLocation {
+    pub frame_no: u32,
+}
+
+impl FrameLocation {
+    pub const IN_MAIN_DB_FILE: u32 = 0;
+
+    pub fn new(frame_no: u32) -> Self {
+        Self { frame_no }
+    }
+
+    pub fn in_wal_file(frame_no: u32) -> Self {
+        assert_ne!(frame_no, FrameLocation::IN_MAIN_DB_FILE);
+        Self { frame_no }
+    }
+
+    pub fn in_main_db_file() -> Self {
+        Self {
+            frame_no: Self::IN_MAIN_DB_FILE,
+        }
+    }
+}
+
+#[repr(C)]
+#[derive(Clone, Copy, Debug, Zeroable, Pod)]
+pub struct FrameRef {
+    pub header: FrameHeader,
+    pub location: FrameLocation,
+    _pad: u32,
+}
+
+impl FrameRef {
+    pub const SIZE: usize = size_of::<Self>();
+
+    pub fn new(header: FrameHeader, location: FrameLocation) -> Self {
+        Self {
+            header,
+            location,
+            _pad: 0,
+        }
+    }
+
+    pub fn as_bytes(&self) -> Bytes {
+        Bytes::copy_from_slice(bytes_of(self))
+    }
+
+    pub fn try_from_bytes(data: Bytes) -> anyhow::Result<Self> {
+        anyhow::ensure!(data.len() == Self::SIZE, "invalid frame size");
+        try_from_bytes(&data)
+            .copied()
+            .map_err(|e| anyhow::anyhow!(e))
+    }
+}
+
 impl fmt::Debug for Frame {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         f.debug_struct("Frame")