Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use PyHive pure-sasl import #818

Merged
merged 35 commits into from
Aug 21, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
d359a8f
Use PyHive pure-sasl import
colin-rogers-dbt Jun 28, 2023
bb3c688
Test PyHive pure-sasl import
colin-rogers-dbt Jun 28, 2023
44f1b14
update requirements.txt
colin-rogers-dbt Jun 28, 2023
7360353
use github fix for >3.11
colin-rogers-dbt Jun 28, 2023
3779c2d
use github fix for >3.11
colin-rogers-dbt Jun 28, 2023
4982589
fix pyhive spec
colin-rogers-dbt Jun 28, 2023
71030e3
remove unnecessary requests extra
colin-rogers-dbt Jun 28, 2023
ddd1f79
remove unnecessary sasl install
colin-rogers-dbt Jun 28, 2023
bcb47f5
add changie
colin-rogers-dbt Jun 28, 2023
b5b2b20
specify no-binary for pyodbc install
colin-rogers-dbt Jun 28, 2023
152a2ee
conditionally import sasl vs puresasl
colin-rogers-dbt Jun 28, 2023
0a51daa
conditionally import sasl vs pure-sasl
colin-rogers-dbt Jun 28, 2023
3c18e5b
Merge branch 'main' into usePureSasl
colin-rogers-dbt Jun 29, 2023
c80e599
experiment with pure sasl SASLClient
colin-rogers-dbt Jun 29, 2023
50a5a54
file formatting
colin-rogers-dbt Jun 29, 2023
9e738b1
test using only puresasl
colin-rogers-dbt Jul 1, 2023
52a2ba7
test using only puresasl
colin-rogers-dbt Jul 1, 2023
a03e364
update requirements.txt
colin-rogers-dbt Jul 1, 2023
7987673
remove sasl from req
colin-rogers-dbt Jul 5, 2023
3941979
remove unused sys dependency
colin-rogers-dbt Jul 5, 2023
6ff75f7
Merge branch 'main' into usePureSasl
colin-rogers-dbt Jul 10, 2023
078d2c6
Merge branch 'main' into usePureSasl
mikealfare Jul 12, 2023
6d6c561
Merge branch 'main' into usePureSasl
mikealfare Jul 19, 2023
ef99d3e
test spark-session
colin-rogers-dbt Aug 1, 2023
645c695
fix pure-sasl
colin-rogers-dbt Aug 1, 2023
86d9767
allow pre-release installs
colin-rogers-dbt Aug 3, 2023
5ea5c55
allow pre-release installs
colin-rogers-dbt Aug 3, 2023
9ea4393
Merge branch 'main' into usePureSasl
colin-rogers-dbt Aug 3, 2023
3486590
Merge branch 'main' into usePureSasl
colin-rogers-dbt Aug 3, 2023
cede12f
use pyhive 0.7.0
colin-rogers-dbt Aug 21, 2023
215b9fa
merge main
colin-rogers-dbt Aug 21, 2023
c88567f
add pyhive back to reqs
colin-rogers-dbt Aug 21, 2023
2851b78
fix setup.py and remove sasl from dev-requirements.txt
colin-rogers-dbt Aug 21, 2023
9e0769e
Update Dependencies-20230628-121341.yaml
colin-rogers-dbt Aug 21, 2023
fc2c610
remove unnecessary pip_pre arg from tox.ini
colin-rogers-dbt Aug 21, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changes/unreleased/Dependencies-20230628-121341.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Dependencies
body: Replace sasl with pure-sasl for PyHive
time: 2023-06-28T12:13:41.141588-07:00
custom:
Author: colin-rogers-dbt
PR: "818"
4 changes: 2 additions & 2 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10"]
python-version: ["3.8", "3.9", "3.10", "3.11"]

env:
TOXENV: "unit"
Expand Down Expand Up @@ -177,7 +177,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ["3.8", "3.9", "3.10"]
python-version: ["3.8", "3.9", "3.10", "3.11"]

steps:
- name: Set up Python ${{ matrix.python-version }}
Expand Down
14 changes: 6 additions & 8 deletions dbt/adapters/spark/connections.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@
from thrift.transport.TSSLSocket import TSSLSocket
import thrift
import ssl
import sasl
import thrift_sasl
from puresasl.client import SASLClient
except ImportError:
pass # done deliberately: setting modules to None explicitly violates MyPy contracts by degrading type semantics

Expand Down Expand Up @@ -603,17 +603,15 @@ def build_ssl_transport(
# to be nonempty.
password = "x"

def sasl_factory() -> sasl.Client:
sasl_client = sasl.Client()
sasl_client.setAttr("host", host)
def sasl_factory() -> SASLClient:
if sasl_auth == "GSSAPI":
sasl_client.setAttr("service", kerberos_service_name)
sasl_client = SASLClient(host, kerberos_service_name, mechanism=sasl_auth)
elif sasl_auth == "PLAIN":
sasl_client.setAttr("username", username)
sasl_client.setAttr("password", password)
sasl_client = SASLClient(
host, mechanism=sasl_auth, username=username, password=password
)
else:
raise AssertionError
sasl_client.init()
return sasl_client

transport = thrift_sasl.TSaslClientTransport(sasl_factory, sasl_auth, socket)
Expand Down
1 change: 0 additions & 1 deletion dev-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,4 @@ wheel~=0.41

# Adapter specific dependencies
mock~=5.1
sasl~=0.3.1
thrift_sasl~=0.4.3
4 changes: 2 additions & 2 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
PyHive[hive]>=0.6.0,<0.7.0
requests[python]>=2.28.1
pyhive[hive_pure_sasl]~=0.7.0
requests>=2.28.1

pyodbc~=4.0.39
sqlparams>=3.0.0
Expand Down
6 changes: 3 additions & 3 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
print("Please upgrade to Python 3.8 or higher.")
sys.exit(1)


# require version of setuptools that supports find_namespace_packages
from setuptools import setup

Expand Down Expand Up @@ -54,9 +53,9 @@ def _get_dbt_core_version():
dbt_core_version = _get_dbt_core_version()
description = """The Apache Spark adapter plugin for dbt"""

odbc_extras = ["pyodbc~=4.0.30"]
odbc_extras = ["pyodbc~=4.0.39"]
pyhive_extras = [
"PyHive[hive]>=0.6.0,<0.7.0",
"PyHive[hive_pure_sasl]~=0.7.0",
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
"thrift>=0.11.0,<0.17.0",
]
session_extras = ["pyspark>=3.0.0,<4.0.0"]
Expand Down Expand Up @@ -93,6 +92,7 @@ def _get_dbt_core_version():
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
],
python_requires=">=3.8",
)
Loading