Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Test] Progressive rendering for big dataset #5

Open
wants to merge 36 commits into
base: master
Choose a base branch
from
Open

[Test] Progressive rendering for big dataset #5

wants to merge 36 commits into from

Conversation

lixun910
Copy link
Member

@lixun910 lixun910 commented Nov 3, 2023 •
edited
Loading

Test progressive rendering based on the batches returned from loaders.gl. This could be used to load a really big dataset.
Base on loaders.gl v4 and geoarrow work.

  • no impact on small or medium-sized dataset
  • instantaneous rendering of the first batch
  • rendering while loading more batches

Note:

  • Arrow: batch size can be set by -lco BATCH_SIZE=65536 (rows)

Loading ~1 million polygons
incremental-render

Loading ~11 million polygons with 1 million batch size
incremental-render-ca-1mi

@lixun910 lixun910 mentioned this pull request Nov 3, 2023
Merged
8 tasks
@lixun910 lixun910 changed the base branch from upstream-xli-geoarrow to master November 15, 2023 05:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lanselin lanselin Awaiting requested review from lanselin

@jkoschinsky jkoschinsky Awaiting requested review from jkoschinsky

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@lixun910