Unable to allocate 161. GiB for array with panicked failed

[heyuqi1970]

What language are you using?

Python 3.8.10

What version are you using?

0.32

What database are you using?

oracle

What dataframe are you using?

Arrow

Can you describe your bug?

What are the steps to reproduce the behavior?

Database setup if the error only happens on specific data or data type

Example query / code

select * from table

What is the error?

numpy.core._exceptions>ArrayMemoryError: Unable to allocate 161. GiB for array with shape <102, 211638219> and data type object
thread '' panicked at 'Python API call failedfailed'

[heyuqi1970]

my system only 4G memory,

[rudyryk]

ConnectorX load all data from the query, so if table contains 160Gb data is tries to allocate such amount of memory.

Looks like there's no chunked downloading capability, what we can do is limit through queries itself or download by ranges by ourselved, e.g.:

select * from table where id > 1000000 and id < 2000000