Nulls are not encoded efficiently

[adriangb]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import os
import polars as pl

df = pl.DataFrame(
    {'a': [1] + [None] * 1_000_000}
)

df.write_parquet('z1.parquet')
print(os.path.getsize('z1.parquet'))  # 125976

df.write_parquet('z2.parquet', use_pyarrow=True)
print(os.path.getsize('z2.parquet'))  # 600

Log output

No response

Issue description

Nulls are not encoded efficiently.

Expected behavior

Output file size matches pyarrow (give or take).

Installed versions

--------Version info---------
Polars:               0.20.25
Index type:           UInt32
Platform:             macOS-14.4.1-arm64-arm-64bit
Python:               3.12.3 (main, Apr  9 2024, 08:09:14) [Clang 15.0.0 (clang-1500.3.9.4)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           <not installed>
deltalake:            0.17.4
fastexcel:            <not installed>
fsspec:               <not installed>
gevent:               <not installed>
hvplot:               <not installed>
matplotlib:           <not installed>
nest_asyncio:         1.6.0
numpy:                1.26.4
openpyxl:             <not installed>
pandas:               2.1.4
pyarrow:              16.0.0
pydantic:             2.7.1
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           2.0.24
torch:                <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>```

</details>

[s-banach]

The file size was smaller in the previous version, polars==0.20.24, which included #15959.
(I get 899 as the size for z1.parquet.)
However, that version had a data corruption bug, so it was reverted.
#16125 should be available in the next polars release.

The file size reduction doesn't really have anything to do with nulls per se, just repeated values.

[ritchie46]

I think we can close this then, as #16125 will be released soon.

[adriangb]

Thanks folks!