Update internal schema representation (ponder-sh#852)

* new schema dir * feat: cleanup intermediate types * wip * wip * fix some tests * update type: * fix: schema encoding into lock table * fix jsdoc * nits * decodeSchema * feat: drop schema from namespace_lock, add table_names * fix: namespace_lock migrations * Revert "fix: namespace_lock migrations" This reverts commit 7a02bfc. * Revert "feat: drop schema from namespace_lock, add table_names" This reverts commit f0d9e2d. * Revert "decodeSchema" This reverts commit 8311a37. * add comments explaining schema encoding * Allow custom indexes to be defined in "ponder.schema.ts" (ponder-sh#854) * custom index * feat: create custom indexes when healthy * feat: create custom when healthy * add jsdoc to p.index * more constraint validation * improve constraint logs * . * fix test flake * initial custom index docs * fix: schema docstring * index ordering * rename addIndexes to createIndexes * feat: index ordering * feat: indexing ordering not available on multi columns * add to custom index docs * update intermediate table type * feat: custom indexes with postgres supports custom schemas * docs * docs + changeset * add index to example --------- Co-authored-by: typedarray <[email protected]> --------- Co-authored-by: typedarray <[email protected]>
jcheese1 · May 7, 2024 · b16ab1a · b16ab1a
1 parent 3415b79
commit b16ab1a
Show file tree

Hide file tree

Showing 51 changed files with 2,859 additions and 1,949 deletions.
diff --git a/.changeset/proud-islands-taste.md b/.changeset/proud-islands-taste.md
@@ -0,0 +1,5 @@
+---
+"@ponder/core": patch
+---
+
+Added support for database indexes using the `p.index()` function in `ponder.schema.ts`. [Read more](https://ponder.sh/docs/schema#indexes).
diff --git a/docs/pages/docs/query/direct-sql.mdx b/docs/pages/docs/query/direct-sql.mdx
@@ -176,6 +176,10 @@ These are the [SQLite data types](https://www.sqlite.org/datatype3.html) used by
 
 SQLite does not natively support integers larger than 8 bytes. To safely store and compare large integers (such as 32-byte EVM `uint256{:solidity}` values) in SQLite, we designed an encoding that uses `VARCHAR(79){:sql}` and takes advantage of SQLite's native lexicographic sort. [Here is the reference implementation](https://github.com/ponder-sh/ponder/blob/main/packages/core/src/utils/encoding.ts) used by Ponder internally.
 
+### Indexes
+
+To create indexes on specific columns, use the `p.index()` function in `ponder.schema.ts` Do not manually construct database indexes. [Read more](/docs/schema#indexes).
+
 ## Postgres
 
 ### Database schema
@@ -328,3 +332,6 @@ These are the [Postgres data types](https://www.postgresql.org/docs/current/data
 | `p.float(){:ts}`   | `FLOAT8{:sql}`/`DOUBLE{:sql}` |                                                           |
 | `p.boolean(){:ts}` | `INTEGER{:sql}`               | `0` is `false{:ts}`, `1` is `true{:ts}`                   |
 
+### Indexes
+
+To create indexes on specific columns, use the `p.index()` function in `ponder.schema.ts` Do not manually construct database indexes. [Read more](/docs/schema#indexes).
diff --git a/docs/pages/docs/query/graphql.mdx b/docs/pages/docs/query/graphql.mdx
@@ -479,6 +479,15 @@ type Person {
 
 </div>
 
+## How to speed up queries
+
+Here are a few tips for speeding up slow queries.
+
+1. **Create database indexes**: Create indexes to speed up filters, joins, and sort conditions. [Read more](/docs/schema#indexes).
+2. **Enable horizontal scaling**: If the GraphQL API is struggling to keep up with reqeust volume, consider spreading the load across multiple instances. [Read more](/docs/production/horizontal-scaling).
+3. **Limit query depth**: Each layer of depth in a GraphQL query introduces at least one additional sequential database query. Avoid queries that are more than 2 layers deep.
+3. **Use pagination**: Use cursor-based pagination to fetch records in smaller, more manageable chunks. This can help reduce the load on the database.
+
 ## Time-travel queries
 
 <Callout type="warning">Native support for time-travel queries was removed in `0.4.0`. [Read more](/docs/indexing/time-series).</Callout>

diff --git a/docs/pages/docs/schema.mdx b/docs/pages/docs/schema.mdx
@@ -525,6 +525,76 @@ query {
 
 </div>
 
+## Indexes
+
+To create a database index, include a second argument to the `p.createTable(){:ts}` function, and use `p.index(){:ts}` to specify which column(s) to include in the index.
+
+<Callout type="info">
+  Database indexes are created _after_ historical indexing is complete, just
+  before the app becomes healthy.
+</Callout>
+
+### Single column
+
+To create a index on a single column, pass the column name to `p.index(){:ts}`.
+
+{/* prettier-ignore */}
+```ts filename="ponder.schema.ts" {10,17}
+import { createSchema } from "@ponder/core";
+
+export default createSchema((p) => ({
+  Person: p.createTable({
+    id: p.string(),
+    name: p.string(),
+    dogs: p.many("Dog.ownerId"),
+  }, {
+    // Index the `name` column to speed up search queries.
+    nameIndex: p.index("name"),
+  }),
+  Dog: p.createTable({
+    id: p.string(),
+    ownerId: p.string().references("Person.id"),
+  }, {
+    // Index the `ownerId` column to speed up relational queries (the `Person.dogs` field).
+    ownerIdIndex: p.index("ownerId"),
+  }),
+}));
+```
+
+For single-column indexes, you can also specify the direction of the index using `.asc(){:ts}` or `.desc(){:ts}`, and the null ordering behavior using `.nullsFirst(){:ts}` or `.nullsLast(){:ts}`. These options can improve query performance if you use the column in an `ORDER BY{:sql}` clause using the same options.
+
+```ts filename="ponder.schema.ts" {6}
+import { createSchema } from "@ponder/core";
+
+export default createSchema((p) => ({
+  Person: p.createTable(
+    { id: p.string(), age: p.string() },
+    { ageIndex: p.index("age").asc().nullsLast() }
+  ),
+}));
+```
+
+### Multiple columns
+
+To create a multi-column index, pass an array of column names to `p.index(){:ts}`. The index will be created using the same column order that you specify.
+
+{/* prettier-ignore */}
+```ts filename="ponder.schema.ts" {9}
+import { createSchema } from "@ponder/core";
+
+export default createSchema((p) => ({
+  Person: p.createTable({
+    id: p.string(),
+    age: p.int(),
+    salary: p.bigint(),
+  }, {
+    seniorityIndex: p.index(["age", "salary"]),
+  }),
+}));
+```
+
+The `.asc(){:ts}`, `.desc(){:ts}`, `.nullsFirst(){:ts}` and `.nullsLast(){:ts}` modifiers are not currently supported for multi-column indexes.
+
 ## ERC20 example
 
 Here's a schema for a simple ERC20 app.

diff --git a/examples/reference-erc20/ponder.schema.ts b/examples/reference-erc20/ponder.schema.ts
@@ -22,17 +22,20 @@ export default createSchema((p) => ({
     owner: p.one("ownerId"),
     spender: p.one("spenderId"),
   }),
-  TransferEvent: p.createTable({
-    id: p.string(),
-    amount: p.bigint(),
-    timestamp: p.int(),
-
-    fromId: p.hex().references("Account.id"),
-    toId: p.hex().references("Account.id"),
-
-    from: p.one("fromId"),
-    to: p.one("toId"),
-  }),
+  TransferEvent: p.createTable(
+    {
+      id: p.string(),
+      amount: p.bigint(),
+      timestamp: p.int(),
+
+      fromId: p.hex().references("Account.id"),
+      toId: p.hex().references("Account.id"),
+
+      from: p.one("fromId"),
+      to: p.one("toId"),
+    },
+    { fromIdIndex: p.index("fromId") },
+  ),
   ApprovalEvent: p.createTable({
     id: p.string(),
     amount: p.bigint(),

diff --git a/packages/core/src/bin/utils/run.ts b/packages/core/src/bin/utils/run.ts
@@ -273,6 +273,8 @@ export async function run({
     }
     await handleFinalize(syncService.finalizedCheckpoint);
 
+    await database.createIndexes({ schema });
+
     server.setHealthy();
     common.logger.info({
       service: "server",

diff --git a/packages/core/src/build/schema.test.ts b/packages/core/src/build/schema.test.ts
@@ -1,6 +1,5 @@
-import { expect, test } from "vitest";
-
 import { createSchema } from "@/schema/schema.js";
+import { expect, test } from "vitest";
 import { safeBuildSchema } from "./schema.js";
 
 test("safeBuildSchema() returns error for duplicate enum values", () => {
@@ -33,6 +32,7 @@ test("safeBuildSchema() returns error for ID column typed as an enum", () => {
     myEnum: p.createEnum(["value1", "value2"]),
     // @ts-expect-error
     myTable: p.createTable({
+      // @ts-expect-error
       id: p.enum("myEnum"),
     }),
   }));
@@ -48,6 +48,7 @@ test("safeBuildSchema() returns error for ID column typed as a 'one' relationshi
   const schema = createSchema((p) => ({
     // @ts-expect-error
     myTable: p.createTable({
+      // @ts-expect-error
       id: p.one("refTableId"),
       refTableId: p.string().references("refTable.id"),
     }),
@@ -67,8 +68,10 @@ test("safeBuildSchema() returns error for ID column typed as a 'many' relationsh
   const schema = createSchema((p) => ({
     // @ts-expect-error
     myTable: p.createTable({
+      // @ts-expect-error
       id: p.many("refTable.myTableId"),
     }),
+    // @ts-expect-error
     refTable: p.createTable({
       id: p.string(),
       myTableId: p.string().references("myTable.id"),
@@ -84,8 +87,9 @@ test("safeBuildSchema() returns error for ID column typed as a 'many' relationsh
 
 test("safeBuildSchema() returns error for ID column with the references modifier", () => {
   const schema = createSchema((p) => ({
-    // @ts-expect-errora
+    // @ts-expect-error
     myTable: p.createTable({
+      // @ts-expect-error
       id: p.string().references("refTable.id"),
     }),
     refTable: p.createTable({
@@ -104,6 +108,7 @@ test("safeBuildSchema() returns error for invalid ID column type boolean", () =>
   const schema = createSchema((p) => ({
     // @ts-expect-error
     myTable: p.createTable({
+      // @ts-expect-error
       id: p.boolean(),
     }),
   }));
@@ -119,6 +124,7 @@ test("safeBuildSchema() returns error for invalid ID column type float", () => {
   const schema = createSchema((p) => ({
     // @ts-expect-error
     myTable: p.createTable({
+      // @ts-expect-error
       id: p.float(),
     }),
   }));
@@ -134,6 +140,7 @@ test("safeBuildSchema() returns error for ID column with optional modifier", ()
   const schema = createSchema((p) => ({
     // @ts-expect-error
     myTable: p.createTable({
+      // @ts-expect-error
       id: p.string().optional(),
     }),
   }));
@@ -149,6 +156,7 @@ test("safeBuildSchema() returns error for ID column with list modifier", () => {
   const schema = createSchema((p) => ({
     // @ts-expect-error
     myTable: p.createTable({
+      // @ts-expect-error
       id: p.string().list(),
     }),
   }));
@@ -195,6 +203,7 @@ test("safeBuildSchema() returns error for 'one' relationship with non-existent r
     // @ts-expect-error
     myTable: p.createTable({
       id: p.string(),
+      // @ts-expect-error
       refColumn: p.one("nonExistentColumn"),
     }),
   }));
@@ -318,3 +327,85 @@ test("safeBuildSchema() returns error for foreign key column type mismatch", ()
     "type does not match the referenced table's ID column type",
   );
 });
+
+test("safeBuildSchema() returns error for empty index", () => {
+  const schema = createSchema((p) => ({
+    myTable: p.createTable(
+      {
+        id: p.string(),
+        col: p.int(),
+      },
+      {
+        colIndex: p.index([]),
+      },
+    ),
+  }));
+
+  const result = safeBuildSchema({ schema });
+  expect(result.status).toBe("error");
+  expect(result.error?.message).toContain("Index 'colIndex' cannot be empty.");
+});
+
+test("safeBuildSchema() returns error for duplicate index", () => {
+  const schema = createSchema((p) => ({
+    myTable: p.createTable(
+      {
+        id: p.string(),
+        col: p.int(),
+      },
+      {
+        colIndex: p.index(["col", "col"]),
+      },
+    ),
+  }));
+
+  const result = safeBuildSchema({ schema });
+  expect(result.status).toBe("error");
+  expect(result.error?.message).toContain(
+    "Index 'colIndex' cannot contain duplicate columns.",
+  );
+});
+
+test("safeBuildSchema() returns error for invalid multi-column index", () => {
+  const schema = createSchema((p) => ({
+    // @ts-expect-error
+    myTable: p.createTable(
+      {
+        id: p.string(),
+        col: p.int(),
+      },
+      {
+        // @ts-expect-error
+        colIndex: p.index(["coll"]),
+      },
+    ),
+  }));
+
+  const result = safeBuildSchema({ schema });
+  expect(result.status).toBe("error");
+  expect(result.error?.message).toContain(
+    "Index 'colIndex' does not reference a valid column.",
+  );
+});
+
+test("safeBuildSchema() returns error for invalid index", () => {
+  const schema = createSchema((p) => ({
+    // @ts-expect-error
+    myTable: p.createTable(
+      {
+        id: p.string(),
+        col: p.int(),
+      },
+      {
+        // @ts-expect-error
+        colIndex: p.index("col1"),
+      },
+    ),
+  }));
+
+  const result = safeBuildSchema({ schema });
+  expect(result.status).toBe("error");
+  expect(result.error?.message).toContain(
+    "Index 'colIndex' does not reference a valid column.",
+  );
+});