Reduce memory usage in parquet format code path

[PatrickJin-db]

This PR contains two optimizations:

1. When converting from arrow to pandas, read batches of rows at a time instead of entire parquet files. This change should eliminate pyarrow overhead almost entirely, which may be up to 40-50% reduction in memory usage.
2. Set split_blocks=False. I believe split_blocks=True was initially introduced in case it might help with memory usage during the arrow to pandas conversion. However, it should be set to false for three reasons:

• In my testing, it doesn't affect the memory usage much (if at all).
• We are converting small batches at a time now, so the memory usage of the conversion is negligible now (expect maybe if the table contains an extremely large number of columns)
• As per the documentation, split_blocks=True may cause pandas consolidations when other operations are done on the dataframe later on. In some of my test tables, this has indeed introduced unnecessary consolidations and actually increased memory usage.

Because change 1 may result in worse performance, this change is gated behind the param convert_in_batches, which is false by default.

Also updated the docstrings of load_as_pandas and load_table_changes_as_pandas.

Tested with unit tests.

[linzhou-db]

the pre-merge failures doesn't seem related to the python changes..

@littlegrasscao could you check if it's related:

 - spark read limit *** FAILED ***
[info]   java.lang.NullPointerException: Cannot invoke "java.io.File.getCanonicalPath()" because the return value of "io.delta.sharing.spark.DeltaSharingSuite.testProfileFile()" is null
[info]   at io.delta.sharing.spark.DeltaSharingSuite.$anonfun$new$94(DeltaSharingSuite.scala:577)
[info]   at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
[info]   at org.scalatest.concurrent.TimeLimits.failAfterImpl(TimeLimits.scala:239)
[info]   at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:230)
[info]   at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:229)
[info]   at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:69)
[info]   at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:155)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)