Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Any limit on the number of CSVs to import? #314

Open
huafen17 opened this issue Feb 11, 2025 · 1 comment
Open

Any limit on the number of CSVs to import? #314

huafen17 opened this issue Feb 11, 2025 · 1 comment

Comments

@huafen17
Copy link

huafen17 commented Feb 11, 2025

General Question
I have about 10k csv files for edges and 16k files for vertices. They are organized under the same directories so for path in the importer config I used wildcard for csv file names. ./data/vertex/*.csv and ../data/edge/*.csv. The importer execution completed successfully, but records loaded was a lot smaller than expected. When I search for "add import source successfully", there's just 2527 occurrences, meaning only 2527 csv files were imported. I wonder if there's any limit on the number of csv files allowed for each import or this is a bug?

@huafen17
Copy link
Author

I later tried repartition the data into bigger CSVs. I ended up with 100 edge files and 250 vertex files, this time all of them were imported. However, when I went further up with the data size (previously 15G, this time 150G), this time with 200 edge files and 500 vertex files, all the vertex files were loaded but only 133 edge files were loaded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant