Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new endpoints to api-server #427

Open
wants to merge 47 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
077a38e
Update for branch-22.08
thomcom Aug 31, 2022
99d76ad
Modify edge mapper to use a table .join, with error handling, instead…
thomcom Sep 1, 2022
7f69939
Remove set_missing_fields_as_nulls workaround.
thomcom Sep 2, 2022
adda810
First bits of template for abstract request handling.
thomcom Sep 2, 2022
e685652
Add routes/gpu
thomcom Sep 2, 2022
5093b86
Add Series.repeat
thomcom Sep 6, 2022
a8f6eb8
Merge branch 'main' into feature/api-server-abstraction
thomcom Oct 5, 2022
c302c01
Fix yarn.lock
thomcom Oct 5, 2022
d4e721f
Clean up and pass a couple of tests.
thomcom Oct 11, 2022
cc3cbab
Adding flexibility to gpu handler.
thomcom Oct 11, 2022
0eca0f4
Fiddling with dynamic dispatch
thomcom Oct 11, 2022
ba4c867
Drop the abstract server.
thomcom Oct 12, 2022
1e1758f
Particles endpoints to send data to client.
thomcom Oct 13, 2022
316a6a6
Fix a couple errors.
thomcom Oct 14, 2022
5288d6c
Refactor particles handler to take two routes.
thomcom Nov 10, 2022
dee37c8
Add lon/lat filtering to request.
thomcom Nov 10, 2022
8e19053
Tweaking for better filtering, memory safety.
thomcom Nov 16, 2022
60c48de
Drop z, w, r, g, b from particles request.
thomcom Nov 17, 2022
e8f6b59
Add logging for timed out currentDataframe.dispose() that causes a 500
thomcom Nov 17, 2022
7e51b88
Pass all tests and add some coverage, particles tests.
thomcom Nov 21, 2022
7401333
Add new test files.
thomcom Nov 21, 2022
4fdc5d7
Write unit tests for particles/get_shader_column in preparation of wr…
thomcom Nov 21, 2022
f28d277
Write npoints test and refactor endpoint.
thomcom Nov 22, 2022
f371cc7
Add quadtree loading route.
thomcom Nov 22, 2022
54d3d31
Add set_polygons_quadtree and refactor gpu_cache.
thomcom Nov 23, 2022
6b2a082
Set poly types.
thomcom Nov 23, 2022
43aef81
Write quadtree/get_points
thomcom Nov 23, 2022
63aee4d
Update modules/demo/api-server/package.json
thomcom Nov 30, 2022
bff76ab
Merge branch 'main' of github.com:rapidsai/node into feature/api-serv…
trxcllnt Nov 30, 2022
6228a61
Update package.json
thomcom Nov 30, 2022
bc7f5bd
Drop repeat
thomcom Nov 30, 2022
8d0f953
Merge branch 'feature/api-server-abstraction' of github.com:thomcom/n…
thomcom Nov 30, 2022
fe8679f
Really important comma needed to go.
thomcom Nov 30, 2022
25ed45c
Merge branch 'feature/api-server-abstraction' of github.com:thomcom/n…
trxcllnt Nov 30, 2022
1be7b71
Update modules/demo/api-server/routes/gpu/index.js
thomcom Jan 4, 2023
f687420
Update modules/demo/api-server/test/routes/graphology.test.js
thomcom Jan 4, 2023
5a5efb5
Update modules/demo/api-server/test/routes/graphology.test.js
thomcom Jan 4, 2023
2d3ccb2
For some reason the route path changed. Upstream changes?
thomcom Jan 3, 2023
7ebef59
Trying to figure out issue with eslint improperly formatting in here.
thomcom Jan 3, 2023
ccb42f8
Reorder exports to make clang-format happy.
thomcom Jan 3, 2023
a537ad2
Writing docs for api-server
thomcom Jan 4, 2023
695832b
Add docs, refactor gpu calls.
thomcom Jan 4, 2023
4c3e0fb
Quadtree tweaks.
thomcom Jan 9, 2023
fa9447c
Add quadtree count endpoint, plus clean up schema.
thomcom Jan 13, 2023
b174028
Endpoint to return just the first n points.
thomcom Jan 17, 2023
6b23188
Next and clear endpoints.
thomcom Jan 18, 2023
230d6a3
Fix bug with error in next endpoint.
thomcom Jan 23, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions modules/cudf/src/column.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ Napi::Function Column::Init(Napi::Env const& env, Napi::Object exports) {
// column/filling.cpp
InstanceMethod<&Column::fill>("fill"),
InstanceMethod<&Column::fill_in_place>("fillInPlace"),
// column/repeat.cpp
InstanceMethod<&Column::repeat>("repeat"),
// column/binaryop.cpp
InstanceMethod<&Column::add>("add"),
InstanceMethod<&Column::sub>("sub"),
Expand Down
10 changes: 10 additions & 0 deletions modules/cudf/src/column.ts
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,16 @@ export interface Column<T extends DataType = any> {
*/
fill(value: Scalar<T>, begin?: number, end?: number, memoryResource?: MemoryResource): Column<T>;

/**
* Repeats the values of this n times.
*
* @param repeats The number of times to repeat the column.
* @param memoryResource The optional MemoryResource used to allocate the result Column's device
* memory.
*/
repeat(value: Scalar<T>, begin?: number, end?: number, memoryResource?: MemoryResource):
Column<T>;

/**
* Fills a range of elements in-place in a column with a scalar value.
*
Expand Down
5 changes: 5 additions & 0 deletions modules/cudf/src/node_cudf/column.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -717,6 +717,11 @@ struct Column : public EnvLocalObjectWrap<Column> {
cudf::scalar const& value,
rmm::mr::device_memory_resource* mr = rmm::mr::get_current_device_resource());

// column/filling/repeat.cpp
Column::wrapper_t repeat(
cudf::size_type repeats,
rmm::mr::device_memory_resource* mr = rmm::mr::get_current_device_resource());
thomcom marked this conversation as resolved.
Show resolved Hide resolved

// column/replace.cpp
Column::wrapper_t replace_nulls(
cudf::column_view const& replacement,
Expand Down
21 changes: 21 additions & 0 deletions modules/cudf/src/series.ts
Original file line number Diff line number Diff line change
Expand Up @@ -811,6 +811,27 @@ export class AbstractSeries<T extends DataType = any> {
this._col.fill(new Scalar({type: this.type, value}), begin, end, memoryResource));
}

/**
* Repeats a Series n times, returning a new Series.
*
* @param repeats The number of times to repeat this.
*
* @example
* ```typescript
* import {Series} from '@rapidsai/cudf';
*
* // Float64Series
* Series.new([1, 2, 3]).repeat(2) // [1, 2, 3, 1, 2, 3]
* // StringSeries
* Series.new(["foo", "bar", "test"]).repeat(2) // ["foo", "bar", "test", "foo", "bar", "test"]
* // Bool8Series
* Series.new([true, true, true]).repeat(2) // [true, false, false, true, false, false]
* ```
*/
repeat(repeats: T['scalarType'], memoryResource?: MemoryResource): Series<T> {
return this.__construct(this._col.repeat(repeats, memoryResource));
thomcom marked this conversation as resolved.
Show resolved Hide resolved
}

/**
* Fills a range of elements in-place in a column with a scalar value.
*
Expand Down
2 changes: 1 addition & 1 deletion modules/demo/api-server/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ const Path = require('path');
// Change cwd to the example dir so relative file paths are resolved
process.chdir(__dirname);

const fastify = require.resolve('fastify-cli/cli.js');
const fastify = require.resolve('fastify-cli/cli.js')({logger: true});

const {spawnSync} = require('child_process');

Expand Down
84 changes: 84 additions & 0 deletions modules/demo/api-server/routes/gpu/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
// Copyright (c) 2022, NVIDIA CORPORATION.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

const Fs = require('fs');
const {Utf8String, Int32, Uint32, Float32, DataFrame, Series, Float64} = require('@rapidsai/cudf');
const {RecordBatchStreamWriter, Field, Vector, List, Table} = require('apache-arrow');
const Path = require('path');
const {promisify} = require('util');
const Stat = promisify(Fs.stat);
const fastifyCors = require('@fastify/cors');
const fastify = require('fastify');

const arrowPlugin = require('fastify-arrow');
const gpu_cache = require('../../util/gpu_cache.js');
const root_schema = require('../../util/schema.js');

module.exports = async function(fastify, opts) {
fastify.register(arrowPlugin);
fastify.register(fastifyCors, {origin: '*'});
fastify.decorate('setDataframe', gpu_cache.setDataframe);
fastify.decorate('getDataframe', gpu_cache.getDataframe);
fastify.decorate('gpu', gpu_cache);

const get_handler =
async (request, reply) => {
const query = request.query;
request.log.info('Parsing Query:');
request.log.info(query);
request.log.info('Sending query to gpu_cache');
request.log.info('Updating result');
request.log.info('Sending cache.tick');
let result = {
'params': JSON.stringify(query),
success: true,
message: `gpu method:${request.method} placeholder`,
statusCode: 200
};
return result
}

const post_handler =
async (request, reply) => {
const query = request.query;
request.log.info('Parsing Query:');
request.log.info(query);
request.log.info('Sending query to gpu_cache');
request.log.info('Updating result');
request.log.info('Sending cache.tick');
let result = {
'params': JSON.stringify(query),
success: true,
message: `gpu method:${request.method} placeholder`,
statusCode: 200
};
return result
}

const get_schema = {
logLevel: 'debug',
schema: {
response: {
200: {
type: 'object',
properties:
{success: {type: 'boolean'}, message: {type: 'string'}, params: {type: 'string'}}
}
}
}
};

fastify.get('/', {...get_schema, handler: get_handler});
fastify.post('/', {...get_schema, handler: post_handler});
}
11 changes: 11 additions & 0 deletions modules/demo/api-server/util/schema.js
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,17 @@
'use strict';

const schema = {
gpu: {
description: 'An abstract interface to the node-rapids api, supported by a server.',
schema: {
'/': {
method: 'The name of the method to apply to gpu_cache data.',
caller: 'Either an object that has been stored in the gpu_cache or a static module name.',
arguments: 'Correctly specified arguments to the gpu_cache method.',
result: 'Either a result code specifying success or failure or an arrow data buffer.',
}
}
},
graphology: {
description: 'The graphology api provides GPU acceleration of graphology datasets.',
schema: {
Expand Down
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,8 @@
"modules/demo/client-server",
"modules/demo/api-server",
"modules/demo/viz-app",
"modules/demo/sql/*"
"modules/demo/sql/*",
"modules/external/attractor-viz-node-rapids/*"
thomcom marked this conversation as resolved.
Show resolved Hide resolved
],
"dependencies": {
"@typescript-eslint/eslint-plugin": "5.30.0",
Expand Down