experimental pglite

docs/pages/docs/api-reference/config.mdx

@@ -197,12 +197,6 @@ export default createConfig({
 
 </details>
 
-## Options
-
-| field                      |   type   |                                                                                                                                                                                                                                  |
-| :------------------------- | :------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **maxHealthcheckDuration** | `number` | **Default: `4 * 60`**. Maximum number of seconds to wait for event processing to be complete before responding as healthy. If event processing takes longer than this amount of time, the GraphQL API may serve incomplete data. |
-
 ## Examples
 
 ### Basic example

docs/pages/docs/production/_meta.ts

@@ -1,5 +1,4 @@
 export default {
   deploy: "Deploy",
-  "zero-downtime": "Zero downtime",
   "horizontal-scaling": "Horizontal scaling",
 };

docs/pages/docs/production/deploy.mdx

@@ -8,6 +8,26 @@ import { Callout } from "nextra/components";
 
 # Deploy
 
+## Database schema
+
+Ponder uses a single database schema for user tables, metadata, and reorg information. This schema follows a strict set of rules to avoid two Ponder apps writing to the same tables or accidentally dropping tables from a running Ponder app.
+
+<Callout type="default">
+  The default schema is "public" but gets [overriden](/docs/production/deploy#zero-downtime-deployments) when executing on Railway. This behavior can be configured in [`ponder.config.ts`](/docs/api-reference/config#postgres)
+</Callout>
+
+## Cached indexing
+
+## Healthchecks
+
+### `/health`
+
+Returns a `200` status code immediately after the app starts running
+
+### `/ready`
+
+Returns a `200` status code after the app has completed the historical backfill and is available to serve traffic
+
 ## Railway (recommended)
 
 [Railway](https://railway.app) is currently the best place to deploy Ponder apps. Most Ponder apps cost ~$5 per month, and the deployment process is simple.
@@ -25,7 +45,7 @@ From the Railway console:
 1. Click **New Project** → **Deploy from GitHub repo** and select your repo from the list
 2. Click **Add variables**, then add your project's RPC URL (e.g. `PONDER_RPC_URL_1`) and any other environment variables
 3. Expose your service to the public internet. Open the **Settings** tab and click **Generate Domain** under **Networking**
-4. Set a healthcheck path. In the **Settings** tab, enter `/health` for **Healthcheck Path** under **Deploy**
+4. Set a healthcheck path. In the **Settings** tab, enter `/ready` for **Healthcheck Path** under **Deploy**
 
 <Callout type="warning">
   _Monorepo users:_ Update the service **Root Directory** and/or **Start Command**
@@ -44,6 +64,16 @@ After a moment, your Ponder service should redeploy successfully. Check the **Bu
 
 </Steps>
 
+### Zero downtime deployments
+
+Ponder uses a unique database schema for each deployment on Railway. By default, this allows for zero downtime deployments with reindexing occurring on every redployment. 
+
+<Callout type="warning">
+  **Healthcheck Timeout** must be set higher than the time it takes to complete the historical backfill.
+</Callout>
+
+### Manual redployments
+
 ## Other environments
 
 Ponder has not been thoroughly tested on cloud providers other than Railway. However, Ponder apps should work in any environment that supports Node.js and can connect to a Postgres database.

docs/pages/docs/production/horizontal-scaling.mdx

@@ -48,7 +48,8 @@ Other than the start command, the `ponder serve` service should have the same co
 1. Click **New** → **GitHub repo** and select your Ponder repo from the list.
 2. Open the **Settings** tab for the new service and update the **Start Command** to `ponder serve`.
 3. Open the **Variables** tab, click on **New Variable** → **Add Reference** and select `DATABASE_URL`. Be sure to use the same database as the `ponder start` service.
-4. Set a healthcheck path. On the **Settings** tab, go to **Deploy** → **Healthcheck Path** and enter `/health`.
+4. [set schema]
+5. Set a healthcheck path. On the **Settings** tab, go to **Deploy** → **Healthcheck Path** and enter `/health`.
 
 ### Create replicas
 

docs/pages/docs/query/api-functions.mdx

@@ -249,6 +249,7 @@ ponder.hono.notFound((c) => {
 
 If you register API functions that conflict with these internal routes, the build will fail.
 
-- `/health`: Returns a `200` status code after the app has completed historical indexing OR the healthcheck timeout has expired, whichever comes first. [Read more](/docs/production/zero-downtime) about healthchecks.
+- `/health`: Returns a `200` status code immediately after the app starts running. [Read more](/docs/production/deploy#healthchecks) about healthchecks.
+- `/ready`: Returns a `200` status code after the app has completed the historical backfill and is available to serve traffic. [Read more](/docs/production/deploy#healthchecks) about heatlthchecks.
 - `/metrics`: Returns Prometheus metrics. [Read more](/docs/advanced/metrics) about metrics.
 - `/status`: Returns indexing status object. [Read more](/docs/advanced/status) about indexing status.

docs/pages/docs/query/direct-sql.mdx

@@ -18,13 +18,11 @@ During local development, Ponder uses SQLite database files located in the `.pon
 <FileTree>
   <FileTree.Folder name=".ponder/sqlite" open>
     <FileTree.File name="public.db" />
-    <FileTree.File name="ponder.db" />
     <FileTree.File name="ponder_sync.db" />
   </FileTree.Folder>
 </FileTree>
 
 - **`public.db`**: Contains a live indexed table for each table in `ponder.schema.ts`. It's safe to read data from this database during local development and testing.
-- **`ponder.db`**: Contains internal tables that support chain reorg reconciliation.
 - **`ponder_sync.db`**: Contains raw blockchain data that has been cached locally.
 
 Note: SQLite sometimes creates a `-shm` and `-wal` file alongside database files. Do not modify or delete them.
@@ -199,17 +197,7 @@ export default createConfig({
 });
 ```
 
-Like with SQLite, Ponder also uses the `ponder` and `ponder_sync` Postgres database schemas for internal tables and cached RPC data.
-
-#### Multiple instances
-
-To run multiple Ponder instances against the same database, each instance **must** use a different database schema.
-
-Before `0.4`, Ponder used a random schema name for each instance. This solved the isolation problem, but made it difficult to know where to look for indexed data, and cluttered the database.
-
-<Callout>
-	When using a platform with zero-downtime deployments like Railway, Ponder uses a unique schema for each deployment. [Read more](/docs/production/zero-downtime#direct-sql-with-zero-downtime) on zero-downtime deployments for direct SQL.
-</Callout>
+Like with SQLite, Ponder also uses the `ponder_sync` Postgres database schemas for cached RPC data.
 
 ### Connect using `psql`
 

packages/core/package.json

@@ -45,13 +45,13 @@
   "dependencies": {
     "@babel/code-frame": "^7.23.4",
     "@commander-js/extra-typings": "^12.0.1",
+    "@electric-sql/pglite": "^0.2.5",
     "@escape.tech/graphql-armor-max-aliases": "^2.3.0",
     "@escape.tech/graphql-armor-max-depth": "^2.2.0",
     "@escape.tech/graphql-armor-max-tokens": "^2.3.0",
     "@hono/node-server": "^1.11.2",
     "@ponder/utils": "workspace:*",
     "abitype": "^0.10.2",
-    "better-sqlite3": "^11.1.2",
     "commander": "^12.0.0",
     "conf": "^12.0.0",
     "dataloader": "^2.2.2",
@@ -64,6 +64,7 @@
     "http-terminator": "^3.2.0",
     "ink": "^4.4.1",
     "kysely": "^0.26.3",
+    "kysely-pglite": "^0.6.0",
     "pg": "^8.11.3",
     "pg-connection-string": "^2.6.2",
     "picocolors": "^1.0.0",
@@ -77,7 +78,6 @@
   },
   "devDependencies": {
     "@types/babel__code-frame": "^7.0.6",
-    "@types/better-sqlite3": "^7.6.10",
     "@types/glob": "^8.1.0",
     "@types/node": "^20.10.0",
     "@types/pg": "^8.10.9",

packages/core/src/_test/setup.ts

@@ -10,14 +10,13 @@ import { createTelemetry } from "@/common/telemetry.js";
 import type { Config } from "@/config/config.js";
 import type { DatabaseConfig } from "@/config/database.js";
 import type { Network } from "@/config/networks.js";
-import { PostgresDatabaseService } from "@/database/postgres/service.js";
-import type { DatabaseService, NamespaceInfo } from "@/database/service.js";
-import { SqliteDatabaseService } from "@/database/sqlite/service.js";
+import { type Database, createDatabase } from "@/database/index.js";
 import { createSchema } from "@/index.js";
 import { getHistoricalStore } from "@/indexing-store/historical.js";
 import { getReadonlyStore } from "@/indexing-store/readonly.js";
 import { getRealtimeStore } from "@/indexing-store/realtime.js";
 import type { IndexingStore, ReadonlyStore } from "@/indexing-store/store.js";
+import type { Schema } from "@/schema/common.js";
 import { type SyncStore, createSyncStore } from "@/sync-store/index.js";
 import type { BlockSource, ContractSource, LogFactory } from "@/sync/source.js";
 import type { RequestQueue } from "@/utils/requestQueue.js";
@@ -114,79 +113,67 @@ export async function setupIsolatedDatabase(context: TestContext) {
   }
 }
 
-type DatabaseServiceSetup = Parameters<DatabaseService["setup"]>[0] & {
+type DatabaseServiceSetup = {
+  buildId: string;
+  schema: Schema;
   indexing: "realtime" | "historical";
 };
-const defaultSchema = createSchema(() => ({}));
 const defaultDatabaseServiceSetup: DatabaseServiceSetup = {
   buildId: "test",
-  schema: defaultSchema,
+  schema: createSchema(() => ({})),
   indexing: "historical",
 };
 
 export async function setupDatabaseServices(
   context: TestContext,
   overrides: Partial<DatabaseServiceSetup> = {},
 ): Promise<{
-  database: DatabaseService;
-  namespaceInfo: NamespaceInfo;
+  database: Database;
   syncStore: SyncStore;
   indexingStore: IndexingStore;
   readonlyStore: ReadonlyStore;
   cleanup: () => Promise<void>;
 }> {
   const config = { ...defaultDatabaseServiceSetup, ...overrides };
-  let database: SqliteDatabaseService | PostgresDatabaseService;
-
-  if (context.databaseConfig.kind === "sqlite") {
-    database = new SqliteDatabaseService({
-      common: context.common,
-      directory: context.databaseConfig.directory,
-    });
-  } else {
-    database = new PostgresDatabaseService({
-      common: context.common,
-      poolConfig: context.databaseConfig.poolConfig,
-      userNamespace: context.databaseConfig.schema,
-    });
-  }
+  const database = createDatabase({
+    common: context.common,
+    databaseConfig: context.databaseConfig,
+    schema: config.schema,
+  });
 
-  const result = await database.setup(config);
+  await database.setup(config);
 
-  await database.migrateSyncStore();
+  await database.migrateSync();
 
   const syncStore = createSyncStore({
     common: context.common,
-    sql: database.kind,
-    db: database.syncDb,
+    dialect: database.dialect,
+    db: database.qb.sync,
   });
 
   const readonlyStore = getReadonlyStore({
-    encoding: database.kind,
+    dialect: database.dialect,
     schema: config.schema,
-    namespaceInfo: result.namespaceInfo,
-    db: database.indexingDb,
+    db: database.qb.user,
     common: context.common,
   });
 
   const indexingStore =
     config.indexing === "historical"
       ? getHistoricalStore({
-          encoding: database.kind,
+          dialect: database.dialect,
           schema: config.schema,
           readonlyStore,
-          namespaceInfo: result.namespaceInfo,
-          db: database.indexingDb,
+          db: database.qb.user,
           common: context.common,
           isCacheExhaustive: true,
         })
       : {
           ...readonlyStore,
           ...getRealtimeStore({
-            encoding: database.kind,
+            dialect: database.dialect,
             schema: config.schema,
-            namespaceInfo: result.namespaceInfo,
-            db: database.indexingDb,
+            db: database.qb.user,
             common: context.common,
           }),
         };
@@ -195,7 +182,6 @@ export async function setupDatabaseServices(
 
   return {
     database,
-    namespaceInfo: result.namespaceInfo,
     readonlyStore,
     indexingStore,
     syncStore,