Edgetest fix (#266)

* updating edgetest config

* adding manual trigger for edgetest

* adding in pyspark edgetest config

.github/workflows/edgetest.yml

@@ -5,6 +5,7 @@ name: Run edgetest
 on:
   schedule:
     - cron: '30 17 * * 5'
+  workflow_dispatch:  # allows manual dispatch
 jobs:
   edgetest:
     runs-on: ubuntu-latest

README.md

@@ -41,7 +41,7 @@ pip install datacompy[ray]
 ### In-scope Spark versions
 Different versions of Spark play nicely with only certain versions of Python below is a matrix of what we test with
 
-|             | Spark 3.1.3  | Spark 3.2.3 | Spark 3.3.4 | Spark 3.4.4 | Spark 3.5.0 |
+|             | Spark 3.1.3  | Spark 3.2.3 | Spark 3.3.4 | Spark 3.4.2 | Spark 3.5.0 |
 |-------------|--------------|-------------|-------------|-------------|-------------|
 | Python 3.8  | ✅           | ✅           | ✅           | ✅          | ✅          |
 | Python 3.9  | ✅           | ✅           | ✅           | ✅          | ✅          |

pyproject.toml

@@ -130,9 +130,10 @@ module = "pyarrow"
 ignore_missing_imports = true
 
 [edgetest.envs.core]
-python_version = "3.9"
+python_version = "3.10"
 conda_install = ["openjdk=8"]
 extras = ["dev"]
+deps = ["pyspark==3.4.2"]
 command = "pytest tests -m 'not integration'"
 upgrade = [
     "pandas",