new feature: Python binding in multiprocessing context

[TennyZhuang]

Feature Description

The operator is stateless, so we have no reason to reject its use in Python's multiprocess environment.

Problem and Solution

Python uses the pickle format to serialize and deserialize Python objects between multiple processes. If it is a native Python object, it is naturally supported to be converted by pickle. However, as an extension, we must implement it manually.

There are two hooks for the purpose, __setstate__ and __getstate__.

However, the two methods means that a Operator must expose all information that can be used to construct it. Currently, the public interface can't do that.

@Zheaoli Can you provide some ideas to achieve the purpose?

Additional Context

Here are examples in polars:

https://github.com/pola-rs/polars/blob/18786acd8d1eb68fc87982b07ce29ecbae0923f0/crates/polars-python/src/lazyframe/serde.rs#L16-L36

Are you willing to contribute to the development of this feature?

• Yes, I am willing to contribute to the development of this feature.

[Xuanwo]

There are two hooks for the purpose, __setstate__ and __getstate__.

We could store the configuration provided by users during the operator build process and reconstruct it when needed.

[TennyZhuang]

There are two hooks for the purpose, __setstate__ and __getstate__.

We could store the configuration provided by users during the operator build process and reconstruct it when needed.

To achieve this, we must add new APIs in the core.

[Xuanwo]

I'm guessing we can save scheme and map at python side:

opendal/bindings/python/src/operator.rs

Line 33 in 41b6f27

let mut op = ocore::Operator::via_iter(scheme, map).map_err(format_pyerr)?;