Memory fast filling

[gsvijayraajaa]

Hi,

I am trying to index a json file ( about 7GB) with 4 threads. The initial split in data to the threads fast fills the memory and the process gets killed automatically. I am running jsonpyes on AWS m4.xlarge instance. 

Regards,
Vijay Raajaa GS

[xros]

@gsvijayraajaa

what's the amount of your memory? Have you tried to use 2 threads? And what's the error log?

[xros]

And verifying data could be very time/memory consuming. You can separate this job by two processes. 1. Verify them somewhere 2. Then import them.

[xros]

@gsvijayraajaa Maybe now you can update jsonpyes to version 1.3.0 to test it.

#25

For memory safety, it's recommended that make some SWAP room before processing big data.