flush data from memory to a file

[andreypanchenko]

Is it possible to implement a feature in the current project that would reach the memory threshold allocated for a specific process, the one that reads from the database serves this memory in a file, then again reaches this specified memory threshold and writes to the file again, and in this way we could retrieve data from odbc using the odbc driver into a file iteratively. Deiteration is the number of reads and writes to one file or to several if it goes beyond the limit of the one on the code that we allocated to this file?

Let's say we have a parameter --batch-size-memory, and we set it to a 256 MB whenever it reaches, we are appending this data to the current file, or make a new one as it declared in --file-size-threshold.

[pacman82]

I think I do not understand. odbc2parquet already retrieves data iteratively. Each the set of rows fetched in each iteration is called a batch. A --batch-size-memory parameter already exists. However it limits the size of the buffer used to transfer the data. Once written into parquet the data will be smaller than the batch size for each batch. Is this what you are looking for?

[andreypanchenko]

Hello Marcus, thanks for your reply.

I want a handle that will limit the RAM consumed by the running program.

Let's say I have a system constraint, and I want to set, let's say, 300MB of RAM for a long-run process.
While data will load from DB to a parquet, I will constantly consume just 300MB for the odbc2parquet instance.

Does that make sense?

[pacman82]

Yeah, but, again, this already exists: --batch-size-memory does that. For the transfer buffer, which usually is the largest block of memory. In addition to that there is the memory your odbc driver uses, and some overhead, so the process will consume somewhat more than that, but it should be bounded.