Skip to content

Commit

Permalink
updated readme
Browse files Browse the repository at this point in the history
  • Loading branch information
proddata committed Feb 21, 2022
1 parent c04a4dd commit c628368
Show file tree
Hide file tree
Showing 2 changed files with 47 additions and 4 deletions.
47 changes: 44 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
# NodeIngestBench

a very simple node.js script to quickly test ingest performance on large CrateDB
clusters.
a simple nodejs script to quickly test ingest performance on CrateDB clusters.

top measured performance with single node.js client : 1 200 000 rows / s
top measured performance with single nodejs client : 1 200 000 rows / s

## new improved version (app.js)

Ubuntu

```sh
apt update && apt upgrade
Expand All @@ -13,3 +16,41 @@ sudo apt-get install -y nodejs
npm install
node app.js --batchsize 20000 --max_rows 1000000 --table "doc.benchmark" --shards 12 --concurrent_request 20
```


```env
CRATE_HOST=localhost
CRATE_USER=admin
CRATE_PASSWORD=secret-password
CRATE_DB=doc
CRATE_PORT=4200
CRATE_SSL=true
```


## example run with app.js

3 nodes (10 vCPUs) CrateDB cluster

```bash
node app.js --batchsize 20000 --max_rows 1000000 --table "doc.benchmark" --shards 12 --concurrent_request 20
-------- Options ---------
{
dropTable: true,
batchsize: 20000,
max_rows: 6000000,
table: 'doc.benchmark',
shards: 6,
concurrent_requests: 20
}
--------------------------
... Buffer updated
-------- Results ---------
Time 25.179 s
Rows 6,020,000 records
Speed 239,088.13 rows per sec
-------- Results ---------
```

=> 2,400,000 Metrics / sec

4 changes: 3 additions & 1 deletion app.js
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,8 @@ function getNewBufferSync() {

async function updateBuffer(message) {
args_buffer = message.args_buffer;
console.log("... Buffer updated");
let progress = (stats.inserts_done * options.batchsize).toLocaleString();
console.log("Buffer updated - sent: ", progress);
}

worker.on("message", updateBuffer);
Expand All @@ -170,4 +171,5 @@ worker.on("exit", code => {
process.on("SIGTERM", function () {
worker.postMessage({ exit: true });
console.log("Finished all requests");
process.exit();
});

0 comments on commit c628368

Please sign in to comment.