An API for using DuckDB in Node.
This is a high-level API meant for applications. It depends on low-level bindings that adhere closely to DuckDB's C API, available separately as @duckdb/duckdb-bindings.
Main differences from duckdb-node
- Native support for Promises; no need for separate duckdb-async wrapper.
- DuckDB-specific API; not based on the SQLite Node API.
- Lossless & efficent support for values of all DuckDB data types.
- Wraps released DuckDB binaries instead of rebuilding DuckDB.
- Built on DuckDB's C API; exposes more functionality.
Some features are not yet complete:
- Binding advanced data types. (Additional DuckDB C API support needed.)
- Appending advanced data types row-by-row. Appending data chunks recommended instead.
- User-defined types & functions. (Support for this was added to the DuckDB C API in v1.1.0.)
- Profiling info (Added in v1.1.0)
- Table description (Added in v1.1.0)
- APIs for Arrow. (This part of the DuckDB C API is deprecated.)
- Linux arm64 (experimental)
- Linux x64
- Mac OS X (Darwin) arm64 (Apple Silicon)
- Mac OS X (Darwin) x64 (Intel)
- Windows (Win32) x64
import duckdb from '@duckdb/node-api';
console.log(duckdb.version());
console.log(duckdb.configurationOptionDescriptions());
import { DuckDBInstance } from '@duckdb/node-api';
Create with an in-memory database:
const instance = await DuckDBInstance.create(':memory:');
Equivalent to the above:
const instance = await DuckDBInstance.create();
Read from and write to a database file, which is created if needed:
const instance = await DuckDBInstance.create('my_duckdb.db');
Set configuration options:
const instance = await DuckDBInstance.create('my_duckdb.db', {
threads: '4'
});
const connection = await instance.connect();
Connections will be disconnected automatically soon after their reference is dropped, but you can also disconnect explicitly if and when you want:
connection.disconnect();
or, equivalently:
connection.close();
const result = await connection.run('from test_all_types()');
const prepared = await connection.prepare('select $1, $2, $3');
prepared.bindVarchar(1, 'duck');
prepared.bindInteger(2, 42);
prepared.bindList(3, listValue([10, 11, 12]), LIST(INTEGER));
const result = await prepared.run();
or:
const prepared = await connection.prepare('select $a, $b, $c');
prepared.bind({
'a': 'duck',
'b': 42,
'c': listValue([10, 11, 12]),
}, {
'a': VARCHAR,
'b': INTEGER,
'c': LIST(INTEGER),
});
const result = await prepared.run();
or even:
const result = await connection.run('select $a, $b, $c', {
'a': 'duck',
'b': 42,
'c': listValue([10, 11, 12]),
}, {
'a': VARCHAR,
'b': INTEGER,
'c': LIST(INTEGER),
});
Unspecified types will be inferred:
const result = await connection.run('select $a, $b, $c', {
'a': 'duck',
'b': 42,
'c': listValue([10, 11, 12]),
});
Streaming results evaluate lazily when rows are read.
const result = await connection.stream('from range(10_000)');
Get column names and types:
const columnNames = result.columnNames();
const columnTypes = result.columnTypes();
Run and read all data:
const reader = await connection.runAndReadAll('from test_all_types()');
const rows = reader.getRows();
// OR: const columns = reader.getColumns();
Stream and read up to (at least) some number of rows:
const reader = await connection.streamAndReadUntil(
'from range(5000)',
1000
);
const rows = reader.getRows();
// rows.length === 2048. (Rows are read in chunks of 2048.)
Read rows incrementally:
const reader = await connection.streamAndRead('from range(5000)');
reader.readUntil(2000);
// reader.currentRowCount === 2048 (Rows are read in chunks of 2048.)
// reader.done === false
reader.readUntil(4000);
// reader.currentRowCount === 4096
// reader.done === false
reader.readUntil(6000);
// reader.currentRowCount === 5000
// reader.done === true
Result data can be retrieved in a variety of forms:
const reader = await connection.runAndReadAll(
'from range(3) select range::int as i, 10 + i as n'
);
const rows = reader.getRows();
// [ [0, 10], [1, 11], [2, 12] ]
const rowObjects = reader.getRowObjects();
// [ { i: 0, n: 10 }, { i: 1, n: 11 }, { i: 2, n: 12 } ]
const columns = reader.getColumns();
// [ [0, 1, 2], [10, 11, 12] ]
const columnsObject = reader.getColumnsObject();
// { i: [0, 1, 2], n: [10, 11, 12] }
By default, data values that cannot be represented as JS primitives
are returned as rich JS objects; see Inspect Data Values
below.
To retrieve data in a form that can be losslessly serialized to JSON,
use the Json
forms of the above result data methods:
const reader = await connection.runAndReadAll(
'from test_all_types() select bigint, date, interval limit 2'
);
const rows = reader.getRowsJson();
// [
// [
// "-9223372036854775808",
// "5877642-06-25 (BC)",
// { "months": 0, "days": 0, "micros": "0" }
// ],
// [
// "9223372036854775807",
// "5881580-07-10",
// { "months": 999, "days": 999, "micros": "999999999" }
// ]
// ]
const rowObjects = reader.getRowObjectsJson();
// [
// {
// "bigint": "-9223372036854775808",
// "date": "5877642-06-25 (BC)",
// "interval": { "months": 0, "days": 0, "micros": "0" }
// },
// {
// "bigint": "9223372036854775807",
// "date": "5881580-07-10",
// "interval": { "months": 999, "days": 999, "micros": "999999999" }
// }
// ]
const columns = reader.getColumnsJson();
// [
// [ "-9223372036854775808", "9223372036854775807" ],
// [ "5877642-06-25 (BC)", "5881580-07-10" ],
// [
// { "months": 0, "days": 0, "micros": "0" },
// { "months": 999, "days": 999, "micros": "999999999" }
// ]
// ]
const columnsObject = reader.getColumnsObjectJson();
// {
// "bigint": [ "-9223372036854775808", "9223372036854775807" ],
// "date": [ "5877642-06-25 (BC)", "5881580-07-10" ],
// "interval": [
// { "months": 0, "days": 0, "micros": "0" },
// { "months": 999, "days": 999, "micros": "999999999" }
// ]
// }
These methods handle nested types as well:
const reader = await connection.runAndReadAll(
'from test_all_types() select int_array, struct, map, "union" limit 2'
);
const rows = reader.getRowsJson();
// [
// [
// [],
// { "a": null, "b": null },
// [],
// { "tag": "name", "value": "Frank" }
// ],
// [
// [ 42, 999, null, null, -42],
// { "a": 42, "b": "🦆🦆🦆🦆🦆🦆" },
// [
// { "key": "key1", "value": "🦆🦆🦆🦆🦆🦆" },
// { "key": "key2", "value": "goose" }
// ],
// { "tag": "age", "value": 5 }
// ]
// ]
const rowObjects = reader.getRowObjectsJson();
// [
// {
// "int_array": [],
// "struct": { "a": null, "b": null },
// "map": [],
// "union": { "tag": "name", "value": "Frank" }
// },
// {
// "int_array": [ 42, 999, null, null, -42 ],
// "struct": { "a": 42, "b": "🦆🦆🦆🦆🦆🦆" },
// "map": [
// { "key": "key1", "value": "🦆🦆🦆🦆🦆🦆" },
// { "key": "key2", "value": "goose" }
// ],
// "union": { "tag": "age", "value": 5 }
// }
// ]
const columns = reader.getColumnsJson();
// [
// [
// [],
// [42, 999, null, null, -42]
// ],
// [
// { "a": null, "b": null },
// { "a": 42, "b": "🦆🦆🦆🦆🦆🦆" }
// ],
// [
// [],
// [
// { "key": "key1", "value": "🦆🦆🦆🦆🦆🦆" },
// { "key": "key2", "value": "goose"}
// ]
// ],
// [
// { "tag": "name", "value": "Frank" },
// { "tag": "age", "value": 5 }
// ]
// ]
const columnsObject = reader.getColumnsObjectJson();
// {
// "int_array": [
// [],
// [42, 999, null, null, -42]
// ],
// "struct": [
// { "a": null, "b": null },
// { "a": 42, "b": "🦆🦆🦆🦆🦆🦆" }
// ],
// "map": [
// [],
// [
// { "key": "key1", "value": "🦆🦆🦆🦆🦆🦆" },
// { "key": "key2", "value": "goose" }
// ]
// ],
// "union": [
// { "tag": "name", "value": "Frank" },
// { "tag": "age", "value": 5 }
// ]
// }
Fetch all chunks:
const chunks = await result.fetchAllChunks();
Fetch one chunk at a time:
const chunks = [];
while (true) {
const chunk = await result.fetchChunk();
// Last chunk will have zero rows.
if (chunk.rowCount === 0) {
break;
}
chunks.push(chunk);
}
For materialized (non-streaming) results, chunks can be read by index:
const rowCount = result.rowCount;
const chunkCount = result.chunkCount;
for (let i = 0; i < chunkCount; i++) {
const chunk = result.getChunk(i);
// ...
}
Get chunk data:
const rows = chunk.getRows();
const rowObjects = chunk.getRowObjects();
const columns = chunk.getColumns();
const columnsObject = chunk.getColumnsObject();
Get chunk data (one value at a time)
const columns = [];
const columnCount = chunk.columnCount;
for (let columnIndex = 0; columnIndex < columnCount; columnIndex++) {
const columnValues = [];
const columnVector = chunk.getColumnVector(columnIndex);
const itemCount = columnVector.itemCount;
for (let itemIndex = 0; itemIndex < itemCount; itemIndex++) {
const value = columnVector.getItem(itemIndex);
columnValues.push(value);
}
columns.push(columnValues);
}
import { DuckDBTypeId } from '@duckdb/node-api';
if (columnType.typeId === DuckDBTypeId.ARRAY) {
const arrayValueType = columnType.valueType;
const arrayLength = columnType.length;
}
if (columnType.typeId === DuckDBTypeId.DECIMAL) {
const decimalWidth = columnType.width;
const decimalScale = columnType.scale;
}
if (columnType.typeId === DuckDBTypeId.ENUM) {
const enumValues = columnType.values;
}
if (columnType.typeId === DuckDBTypeId.LIST) {
const listValueType = columnType.valueType;
}
if (columnType.typeId === DuckDBTypeId.MAP) {
const mapKeyType = columnType.keyType;
const mapValueType = columnType.valueType;
}
if (columnType.typeId === DuckDBTypeId.STRUCT) {
const structEntryNames = columnType.names;
const structEntryTypes = columnType.valueTypes;
}
if (columnType.typeId === DuckDBTypeId.UNION) {
const unionMemberTags = columnType.memberTags;
const unionMemberTypes = columnType.memberTypes;
}
// For the JSON type (https://duckdb.org/docs/data/json/json_type)
if (columnType.alias === 'JSON') {
const json = JSON.parse(columnValue);
}
Every type implements toString. The result is both human-friendly and readable by DuckDB in an appropriate expression.
const typeString = columnType.toString();
import { DuckDBTypeId } from '@duckdb/node-api';
if (columnType.typeId === DuckDBTypeId.ARRAY) {
const arrayItems = columnValue.items; // array of values
const arrayString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.BIT) {
const bools = columnValue.toBools(); // array of booleans
const bits = columnValue.toBits(); // arrary of 0s and 1s
const bitString = columnValue.toString(); // string of '0's and '1's
}
if (columnType.typeId === DuckDBTypeId.BLOB) {
const blobBytes = columnValue.bytes; // Uint8Array
const blobString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.DATE) {
const dateDays = columnValue.days;
const dateString = columnValue.toString();
const { year, month, day } = columnValue.toParts();
}
if (columnType.typeId === DuckDBTypeId.DECIMAL) {
const decimalWidth = columnValue.width;
const decimalScale = columnValue.scale;
// Scaled-up value. Represented number is value/(10^scale).
const decimalValue = columnValue.value; // bigint
const decimalString = columnValue.toString();
const decimalDouble = columnValue.toDouble();
}
if (columnType.typeId === DuckDBTypeId.INTERVAL) {
const intervalMonths = columnValue.months;
const intervalDays = columnValue.days;
const intervalMicros = columnValue.micros; // bigint
const intervalString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.LIST) {
const listItems = columnValue.items; // array of values
const listString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.MAP) {
const mapEntries = columnValue.entries; // array of { key, value }
const mapString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.STRUCT) {
// { name1: value1, name2: value2, ... }
const structEntries = columnValue.entries;
const structString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.TIMESTAMP_MS) {
const timestampMillis = columnValue.milliseconds; // bigint
const timestampMillisString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.TIMESTAMP_NS) {
const timestampNanos = columnValue.nanoseconds; // bigint
const timestampNanosString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.TIMESTAMP_S) {
const timestampSecs = columnValue.seconds; // bigint
const timestampSecsString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.TIMESTAMP_TZ) {
const timestampTZMicros = columnValue.micros; // bigint
const timestampTZString = columnValue.toString();
const {
date: { year, month, day },
time: { hour, min, sec, micros },
} = columnValue.toParts();
}
if (columnType.typeId === DuckDBTypeId.TIMESTAMP) {
const timestampMicros = columnValue.micros; // bigint
const timestampString = columnValue.toString();
const {
date: { year, month, day },
time: { hour, min, sec, micros },
} = columnValue.toParts();
}
if (columnType.typeId === DuckDBTypeId.TIME_TZ) {
const timeTZMicros = columnValue.micros; // bigint
const timeTZOffset = columnValue.offset;
const timeTZString = columnValue.toString();
const {
time: { hour, min, sec, micros },
offset,
} = columnValue.toParts();
}
if (columnType.typeId === DuckDBTypeId.TIME) {
const timeMicros = columnValue.micros; // bigint
const timeString = columnValue.toString();
const { hour, min, sec, micros } = columnValue.toParts();
}
if (columnType.typeId === DuckDBTypeId.UNION) {
const unionTag = columnValue.tag;
const unionValue = columnValue.value;
const unionValueString = columnValue.toString();
}
if (columnType.typeId === DuckDBTypeId.UUID) {
const uuidHugeint = columnValue.hugeint; // bigint
const uuidString = columnValue.toString();
}
// other possible values are: null, boolean, number, bigint, or string
Converting a TIMESTAMP_TZ value to a string depends on a timezone offset. By default, this is set to the offset for the local timezone when the Node process is started.
To change it, set the timezoneOffsetInMinutes
property of DuckDBTimestampTZValue
:
DuckDBTimestampTZValue.timezoneOffsetInMinutes = -8 * 60;
const pst = DuckDBTimestampTZValue.Epoch.toString();
// 1969-12-31 16:00:00-08
DuckDBTimestampTZValue.timezoneOffsetInMinutes = +1 * 60;
const cet = DuckDBTimestampTZValue.Epoch.toString();
// 1970-01-01 01:00:00+01
Note that the timezone offset used for this string
conversion is distinct from the TimeZone
setting of DuckDB.
The following sets this offset to match the TimeZone
setting of DuckDB:
const reader = await connection.runAndReadAll(
`select (timezone(current_timestamp) / 60)::int`
);
DuckDBTimestampTZValue.timezoneOffsetInMinutes =
reader.getColumns()[0][0];
await connection.run(
`create or replace table target_table(i integer, v varchar)`
);
const appender = await connection.createAppender('main', 'target_table');
appender.appendInteger(42);
appender.appendVarchar('duck');
appender.endRow();
appender.appendInteger(123);
appender.appendVarchar('mallard');
appender.endRow();
appender.flush();
appender.appendInteger(17);
appender.appendVarchar('goose');
appender.endRow();
appender.close(); // also flushes
await connection.run(
`create or replace table target_table(i integer, v varchar)`
);
const appender = await connection.createAppender('main', 'target_table');
const chunk = DuckDBDataChunk.create([INTEGER, VARCHAR]);
chunk.setColumns([
[42, 123, 17],
['duck', 'mallad', 'goose'],
]);
// OR:
// chunk.setRows([
// [42, 'duck'],
// [123, 'mallard'],
// [17, 'goose'],
// ]);
appender.appendDataChunk(chunk);
appender.flush();
const extractedStatements = await connection.extractStatements(`
create or replace table numbers as from range(?);
from numbers where range < ?;
drop table numbers;
`);
const parameterValues = [10, 7];
const statementCount = extractedStatements.count;
for (let stmtIndex = 0; stmtIndex < statementCount; stmtIndex++) {
const prepared = await extractedStatements.prepare(stmtIndex);
let parameterCount = prepared.parameterCount;
for (let paramIndex = 1; paramIndex <= parameterCount; paramIndex++) {
prepared.bindInteger(paramIndex, parameterValues.shift());
}
const result = await prepared.run();
// ...
}
import { DuckDBPendingResultState } from '@duckdb/node-api';
async function sleep(ms) {
return new Promise((resolve) => {
setTimeout(resolve, ms);
});
}
const prepared = await connection.prepare('from range(10_000_000)');
const pending = prepared.start();
while (pending.runTask() !== DuckDBPendingResultState.RESULT_READY) {
console.log('not ready');
await sleep(1);
}
console.log('ready');
const result = await pending.getResult();
// ...
// Run to completion but don't yet retrieve any rows.
// Optionally take values to bind to SQL parameters,
// and (optionally) types of those parameters,
// either as an array (for positional parameters),
// or an object keyed by parameter name.
const result = await connection.run(sql);
const result = await connection.run(sql, values);
const result = await connection.run(sql, values, types);
// Run to completion but don't yet retrieve any rows.
// Wrap in a DuckDBDataReader for convenient data retrieval.
const reader = await connection.runAndRead(sql);
const reader = await connection.runAndRead(sql, values);
const reader = await connection.runAndRead(sql, values, types);
// Run to completion, wrap in a reader, and read all rows.
const reader = await connection.runAndReadAll(sql);
const reader = await connection.runAndReadAll(sql, values);
const reader = await connection.runAndReadAll(sql, values, types);
// Run to completion, wrap in a reader, and read at least
// the given number of rows. (Rows are read in chunks, so more than
// the target may be read.)
const reader = await connection.runAndReadUntil(sql, targetRowCount);
const reader =
await connection.runAndReadAll(sql, targetRowCount, values);
const reader =
await connection.runAndReadAll(sql, targetRowCount, values, types);
// Create a streaming result and don't yet retrieve any rows.
const result = await connection.stream(sql);
const result = await connection.stream(sql, values);
const result = await connection.stream(sql, values, types);
// Create a streaming result and don't yet retrieve any rows.
// Wrap in a DuckDBDataReader for convenient data retrieval.
const reader = await connection.streamAndRead(sql);
const reader = await connection.streamAndRead(sql, values);
const reader = await connection.streamAndRead(sql, values, types);
// Create a streaming result, wrap in a reader, and read all rows.
const reader = await connection.streamAndReadAll(sql);
const reader = await connection.streamAndReadAll(sql, values);
const reader = await connection.streamAndReadAll(sql, values, types);
// Create a streaming result, wrap in a reader, and read at least
// the given number of rows.
const reader = await connection.streamAndReadUntil(sql, targetRowCount);
const reader =
await connection.streamAndReadUntil(sql, targetRowCount, values);
const reader =
await connection.streamAndReadUntil(sql, targetRowCount, values, types);
// Prepared Statements
// Prepare a possibly-parametered SQL statement to run later.
const prepared = await connection.prepare(sql);
// Bind values to the parameters.
prepared.bind(values);
prepared.bind(values, types);
// Run the prepared statement. These mirror the methods on the connection.
const result = prepared.run();
const reader = prepared.runAndRead();
const reader = prepared.runAndReadAll();
const reader = prepared.runAndReadUntil(targetRowCount);
const result = prepared.stream();
const reader = prepared.streamAndRead();
const reader = prepared.streamAndReadAll();
const reader = prepared.streamAndReadUntil(targetRowCount);
// Pending Results
// Create a pending result.
const pending = await connection.start(sql);
const pending = await connection.start(sql, values);
const pending = await connection.start(sql, values, types);
// Create a pending, streaming result.
const pending = await connection.startStream(sql);
const pending = await connection.startStream(sql, values);
const pending = await connection.startStream(sql, values, types);
// Create a pending result from a prepared statement.
const pending = await prepared.start();
const pending = await prepared.startStream();
while (pending.runTask() !== DuckDBPendingResultState.RESULT_READY) {
// optionally sleep or do other work between tasks
}
// Retrieve the result. If not yet READY, will run until it is.
const result = await pending.getResult();
const reader = await pending.read();
const reader = await pending.readAll();
const reader = await pending.readUntil(targetRowCount);
// From a result
// Asynchronously retrieve data for all rows:
const columns = await result.getColumns();
const columnsJson = await result.getColumnsJson();
const columnsObject = await result.getColumnsObject();
const columnsObjectJson = await result.getColumnsObjectJson();
const rows = await result.getRows();
const rowsJson = await result.getRowsJson();
const rowObjects = await result.getRowObjects();
const rowObjectsJson = await result.getRowObjectsJson();
// From a reader
// First, (asynchronously) read some rows:
await reader.readAll();
// or:
await reader.readUntil(targetRowCount);
// Then, (synchronously) get result data for the rows read:
const columns = reader.getColumns();
const columnsJson = reader.getColumnsJson();
const columnsObject = reader.getColumnsObject();
const columnsObjectJson = reader.getColumnsObjectJson();
const rows = reader.getRows();
const rowsJson = reader.getRowsJson();
const rowObjects = reader.getRowObjects();
const rowObjectsJson = reader.getRowObjectsJson();
// Individual values can also be read directly:
const value = reader.value(columnIndex, rowIndex);
// Using chunks
// If desired, one or more chunks can be fetched from a result:
const chunk = await result.fetchChunk();
const chunks = await result.fetchAllChunks();
// And then data can be retrieved from each chunk:
const columnValues = chunk.getColumnValues(columnIndex);
const columns = chunk.getColumns();
const rowValues = chunk.getRowValues(rowIndex);
const rows = chunk.getRows();
// Or, values can be visited:
chunk.visitColumnValues(columnIndex,
(value, rowIndex, columnIndex, type) => { /* ... */ }
);
chunk.visitColumns((column, columnIndex, type) => { /* ... */ });
chunk.visitColumnMajor(
(value, rowIndex, columnIndex, type) => { /* ... */ }
);
chunk.visitRowValues(rowIndex,
(value, rowIndex, columnIndex, type) => { /* ... */ }
);
chunk.visitRows((row, rowIndex) => { /* ... */ });
chunk.visitRowMajor(
(value, rowIndex, columnIndex, type) => { /* ... */ }
);
// Or converted:
// The `converter` argument implements `DuckDBValueConverter`,
// which has the single method convertValue(value, type).
const columnValues = chunk.convertColumnValues(columnIndex, converter);
const columns = chunk.convertColumns(converter);
const rowValues = chunk.convertRowValues(rowIndex, converter);
const rows = chunk.convertRows(converter);
// The reader abstracts these low-level chunk manipulations
// and is recommended for most cases.