[Enh]: Add support for dask

[benrutter]

We would like to learn about your use case. For example, if this feature is needed to adopt Narwhals in an open source project, could you please enter the link to it below?

No response

Please describe the purpose of the new feature or describe the problem to solve.

Dask is a dataframe library built on top of pandas to allow distributed computation of dataframes (like pyspark for example). It's API is a strict subset of the pandas API so in theory integrating it would be feasible. I'd be really down for contributing / looking into how support might work, but wanted to check if it's within the scope/aims of Narwhals or if it's been considered before? (Can't see anything on the issue tracker so I guess no?)

Suggest a solution if possible.

Potentially a lot of pandas code could be reused although I haven't looked in depth to say that which much confidence.

If you have tried alternatives, please describe them below.

No response

Additional information that may help us understand your needs.

I'd love to put in some contributions if it's within scope, I'm pretty experienced with Dask (have made some minor constibutions to dask-dataframe) but not so much with Narwhals. Would be great to get sense of interest for this as a feature, I realise adding in support for a whole new dataframe library is a big undertaking.

[benrutter]

Also there's nowhere in the form to say, but thanks all Narhwalers for putting together such a great library!

[MarcoGorelli]

thanks @benrutter for the request! I'd opened an issue about it here #272, but yours is more complete, so I'll close that one

[MarcoGorelli]

regarding scope - this is definitely a library we need to aim to support

we have a narwhals/_pandas_like folder, where there are implementations for pandas-like APIs (pandas, modin, cudf)

for contributing, a good place to start might be to add a dask dataframe constructor to

narwhals/tests/conftest.py

Lines 34 to 66 in bce5efa

	def pandas_constructor(obj: Any) -> IntoDataFrame:
	return pd.DataFrame(obj) # type: ignore[no-any-return]
	def pandas_nullable_constructor(obj: Any) -> IntoDataFrame:
	return pd.DataFrame(obj).convert_dtypes() # type: ignore[no-any-return]
	def pandas_pyarrow_constructor(obj: Any) -> IntoDataFrame:
	return pd.DataFrame(obj).convert_dtypes(dtype_backend="pyarrow") # type: ignore[no-any-return]
	def modin_constructor(obj: Any) -> IntoDataFrame: # pragma: no cover
	mpd = get_modin()
	return mpd.DataFrame(obj).convert_dtypes(dtype_backend="pyarrow") # type: ignore[no-any-return]
	def polars_constructor(obj: Any) -> IntoDataFrame:
	return pl.DataFrame(obj)
	if parse_version(pd.__version__) >= parse_version("1.5.0"):
	params = [pandas_constructor, pandas_nullable_constructor, pandas_pyarrow_constructor]
	else: # pragma: no cover
	params = [pandas_constructor]
	params.append(polars_constructor)
	if get_modin() is not None: # pragma: no cover
	params.append(modin_constructor)
	@pytest.fixture(params=params)
	def constructor(request: Any) -> Callable[[Any], IntoDataFrame]:
	return request.param # type: ignore[no-any-return]

, then run the test suite, and let the error messages guide you as to what needs fixing. fancy giving this a go?

we also have a discord where you can ask questions whilst contributing if you like, check the readme for the link

[benrutter]

Would absolutely love to give this a go! I'll try to dive into it over the next few days 🙌

[MarcoGorelli]

Dask support is still absolutely welcome, I'll just close in favour of #566

[benrutter]

I'm guessing you mean #566? (this one is #396) Probably a good call, I enjoyed playing around with what support for Dask might look like, but it felt like currently it adds a bunch maintenance difficulties where someone editing the Pandas stuff has to have a good understanding of Dask not to break that too!

[MarcoGorelli]

thanks Ben, that's right!

yeah I think we may need a separate narwhals/_dask folder unfortunately..

[benrutter]

No worries! I'll close off my PR for now but keep tinkering around with thingsin thw fork so that there's a good base for when lazy only is an option.