Code

feat: Add minimal PySpark support#908

Code

Merged

MarcoGorelli merged 98 commits intonarwhals-dev:mainnarwhals-dev/narwhals:mainfrom EdAbati:pysparkEdAbati/narwhals:pysparkCopy head branch name to clipboardDec 5, 2024

+1,375-16

Commits

Commits on Sep 3, 2024

• first pyspark draft

  EdAbati

  committed
  72c1b49

  Copy full SHA for 72c1b49

• Merge remote-tracking branch 'upstream/main' into pyspark

  EdAbati

  committed
  e67140a

  Copy full SHA for e67140a

Commits on Sep 4, 2024

• added schema

  EdAbati

  committed
  3316460

  Copy full SHA for 3316460

• add methods needed for compliant types

  EdAbati

  committed
  12f62c1

  Copy full SHA for 12f62c1

Commits on Sep 7, 2024

• fix all_horizontal

  EdAbati

  committed
  2b114eb

  Copy full SHA for 2b114eb

Commits on Sep 8, 2024

• add xfail to some tests

  EdAbati

  committed
  378b421

  Copy full SHA for 378b421

• draft with sql

  EdAbati

  committed
  b5957dc

  Copy full SHA for b5957dc

Commits on Sep 10, 2024

• merge upstream

  EdAbati

  committed
  9f8f944

  Copy full SHA for 9f8f944

Commits on Sep 11, 2024

• making all frame tests pass

  EdAbati

  committed
  b2aee0e

  Copy full SHA for b2aee0e

Commits on Sep 12, 2024

• group by

  EdAbati

  committed
  0e4b2f2

  Copy full SHA for 0e4b2f2

• skipping tests

  EdAbati

  committed
  741cdde

  Copy full SHA for 741cdde

• restore type

  EdAbati

  committed
  2bdfe31

  Copy full SHA for 2bdfe31

• smaller diff + mypy fix

  EdAbati

  committed
  c0b1a18

  Copy full SHA for c0b1a18

• remove print

  EdAbati

  committed
  ec0b26f

  Copy full SHA for ec0b26f

• Merge remote-tracking branch 'upstream/main' into pyspark

  EdAbati

  committed
  32b87a3

  Copy full SHA for 32b87a3

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for a053b07
  pre-commit-ci[bot]

  committed
  a053b07

  Copy full SHA for a053b07

• smaller diff

  EdAbati

  committed
  a415bd0

  Copy full SHA for a415bd0

• reenable pyspark

  EdAbati

  committed
  6065eb2

  Copy full SHA for 6065eb2

Commits on Oct 6, 2024

• count without window

  EdAbati

  committed
  1688f7d

  Copy full SHA for 1688f7d

• revert expr series tests

  EdAbati

  committed
  191dcb7

  Copy full SHA for 191dcb7

• revert rest of tests

  EdAbati

  committed
  41368ef

  Copy full SHA for 41368ef

• placeholder pyspark test

  EdAbati

  committed
  b0dffad

  Copy full SHA for b0dffad

• merge main

  EdAbati

  committed
  37ecc70

  Copy full SHA for 37ecc70

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 1c76b0b
  pre-commit-ci[bot]

  committed
  1c76b0b

  Copy full SHA for 1c76b0b

• moved test_column

  EdAbati

  committed
  9802fdc

  Copy full SHA for 9802fdc

• moved select filter and with_columns

  EdAbati

  committed
  267f2ff

  Copy full SHA for 267f2ff

• add schema head sort tests

  EdAbati

  committed
  8adee30

  Copy full SHA for 8adee30

• add test add

  EdAbati

  committed
  9186687

  Copy full SHA for 9186687

Commits on Oct 8, 2024

• fix rename

  EdAbati

  committed
  38d326d

  Copy full SHA for 38d326d

• added more tests

  EdAbati

  committed
  223ea88

  Copy full SHA for 223ea88

Commits on Oct 13, 2024

• Merge remote-tracking branch 'upstream/main' into pyspark

  EdAbati

  committed
  3337014

  Copy full SHA for 3337014

• fix all_horizontal

  EdAbati

  committed
  d7b2752

  Copy full SHA for d7b2752

• fixing all tests 🎉🎉

  EdAbati

  committed
  95b8395

  Copy full SHA for 95b8395

• rename test

  EdAbati

  committed
  734c140

  Copy full SHA for 734c140

• add backend_version

  EdAbati

  committed
  1b9a7e7

  Copy full SHA for 1b9a7e7

• added group by tests

  EdAbati

  committed
  9d326a4

  Copy full SHA for 9d326a4

• add pyspark in requirement dev

  EdAbati

  committed
  1a2e804

  Copy full SHA for 1a2e804

• use pyspark.sql to create empty df

  EdAbati

  committed
  411f67d

  Copy full SHA for 411f67d

• stddev for older pyspark

  EdAbati

  committed
  3a59240

  Copy full SHA for 3a59240

• coverage up

  EdAbati

  committed
  08120da

  Copy full SHA for 08120da

Commits on Oct 14, 2024

• min pyspark version test

  EdAbati

  committed
  177ec5e

  Copy full SHA for 177ec5e

• fix for pyspark 3.2

  EdAbati

  committed
  77e6687

  Copy full SHA for 77e6687

• pyspark 3.3 as minimum

  EdAbati

  committed
  9ccab80

  Copy full SHA for 9ccab80

• trying debugging windows

  EdAbati

  committed
  ef1944c

  Copy full SHA for ef1944c

• no test pyspark with pandas <1.0.5

  EdAbati

  committed
  a8b228f

  Copy full SHA for a8b228f

• removing debug windows

  EdAbati

  committed
  c74772d

  Copy full SHA for c74772d

• Merge remote-tracking branch 'upstream/main' into pyspark

  EdAbati

  committed
  dd0dd39

  Copy full SHA for dd0dd39

• testing 3.3.0

  EdAbati

  committed
  d00a2da

  Copy full SHA for d00a2da

• trying with repartition 2

  EdAbati

  committed
  6b25971

  Copy full SHA for 6b25971

Commits on Oct 15, 2024

• remove unused data

  EdAbati

  committed
  3713a6d

  Copy full SHA for 3713a6d

• trying to fix sorting problems in tests

  EdAbati

  committed
  eb0a2ce

  Copy full SHA for eb0a2ce

• no pyspark in minimum_versions

  EdAbati

  committed
  df1a37f

  Copy full SHA for df1a37f

• trying to make windows happy

  EdAbati

  committed
  ce503fa

  Copy full SHA for ce503fa

• fix repartition

  EdAbati

  committed
  94656b3

  Copy full SHA for 94656b3

• exclude pyspark for python 3.12

  EdAbati

  committed
  33739de

  Copy full SHA for 33739de

Commits on Oct 27, 2024

• Merge remote-tracking branch 'upstream/main' into pyspark

  EdAbati

  committed
  5808d71

  Copy full SHA for 5808d71

• use assert_equal_data

  EdAbati

  committed
  5d4b02f

  Copy full SHA for 5d4b02f

• only use self._native_frame.sparkSession

  EdAbati

  committed
  92617f1

  Copy full SHA for 92617f1

• add drop_null_keys in groupby

  EdAbati

  committed
  5733069

  Copy full SHA for 5733069

• rename _spark

  EdAbati

  committed
  9b6c4e0

  Copy full SHA for 9b6c4e0

• rename spark_test

  EdAbati

  committed
  e2344c7

  Copy full SHA for e2344c7

Commits on Oct 28, 2024

• use PYSPARK_VERSION

  EdAbati

  committed
  bb1de48

  Copy full SHA for bb1de48

• rename PySpark... classes to Spark...

  EdAbati

  committed
  36d0886

  Copy full SHA for 36d0886

• _ in func signature

  EdAbati

  committed
  24676d0

  Copy full SHA for 24676d0

• make coverage happy

  EdAbati

  committed
  3defa39

  Copy full SHA for 3defa39

Commits on Nov 17, 2024

• exception public

  EdAbati

  committed
  a8946f2

  Copy full SHA for a8946f2

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 86c459d
  pre-commit-ci[bot]

  committed
  86c459d

  Copy full SHA for 86c459d

• fix docs

  EdAbati

  committed
  dc7fb71

  Copy full SHA for dc7fb71

• Merge remote-tracking branch 'upstream/main' into pyspark

  EdAbati

  committed
  d720b58

  Copy full SHA for d720b58

• rename to _spark_like

  EdAbati

  committed
  a1141f7

  Copy full SHA for a1141f7

• rename exceptions

  EdAbati

  committed
  94b6777

  Copy full SHA for 94b6777

• update coverage to ignore _spark_like

  EdAbati

  committed
  10c1b11

  Copy full SHA for 10c1b11

Commits on Nov 18, 2024

• better comment

  EdAbati

  committed
  5193bca

  Copy full SHA for 5193bca

• invalidintoexpr error

  EdAbati

  committed
  7b513f4

  Copy full SHA for 7b513f4

Commits on Nov 21, 2024

• fix pytest warning error

  EdAbati

  committed
  f25969b

  Copy full SHA for f25969b

• small comment

  EdAbati

  committed
  08987cf

  Copy full SHA for 08987cf

• fix F.std for ddof more than 1

  EdAbati

  committed
  0fb0478

  Copy full SHA for 0fb0478

• Merge remote-tracking branch 'upstream/main' into pyspark

  EdAbati

  committed
  50e2e4d

  Copy full SHA for 50e2e4d

• fix stddev imports for py <3.5

  EdAbati

  committed
  e86fc5c

  Copy full SHA for e86fc5c

• use F

  EdAbati

  committed
  522a1aa

  Copy full SHA for 522a1aa

Commits on Dec 3, 2024

• Merge remote-tracking branch 'upstream/main' into pyspark

  EdAbati

  committed
  84d5b6a

  Copy full SHA for 84d5b6a

• update to latest changes

  EdAbati

  committed
  b9c21df

  Copy full SHA for b9c21df

• add implementation to expr

  EdAbati

  committed
  bb82020

  Copy full SHA for bb82020

• rename SparkLike...

  EdAbati

  committed
  010a362

  Copy full SHA for 010a362

• rename native_to_narwhals_dtype

  EdAbati

  committed
  dac5901

  Copy full SHA for dac5901

• dtype unknown for decimal

  EdAbati

  committed
  6d67b0c

  Copy full SHA for 6d67b0c

• simplify return unknown

  EdAbati

  committed
  15ca58e

  Copy full SHA for 15ca58e

Commits on Dec 4, 2024

• update no_imports_tests

  EdAbati

  committed
  ce4e2fb

  Copy full SHA for ce4e2fb

• level lazy for spark

  EdAbati

  committed
  d841ec5

  Copy full SHA for d841ec5

• add _change_dtypes

  EdAbati

  committed
  ac68a7e

  Copy full SHA for ac68a7e

• Merge remote-tracking branch 'upstream/main' into pyspark

  EdAbati

  committed
  9a1f741

  Copy full SHA for 9a1f741

• _change_version is back

  EdAbati

  committed
  2121c40

  Copy full SHA for 2121c40

• fix no imports tests

  EdAbati

  committed
  c0f44b6

  Copy full SHA for c0f44b6

• rename spark_like tests

  EdAbati

  committed
  4b7895f

  Copy full SHA for 4b7895f

• same error message as dask

  EdAbati

  committed
  638c402

  Copy full SHA for 638c402

Commits on Dec 5, 2024

• remove extra expr._call

  EdAbati

  committed
  b46f1b5

  Copy full SHA for b46f1b5

• update coverage

  EdAbati

  committed
  a3e3dba

  Copy full SHA for a3e3dba

• extract _columns_from_expr

  EdAbati

  committed
  d8e6064

  Copy full SHA for d8e6064