Skip to content

Commit

Permalink
Chore: Remove legacy CI/CD workflow
Browse files Browse the repository at this point in the history
Signed-off-by: Matthew Watkins <[email protected]>
  • Loading branch information
ModeSevenIndustrialSolutions committed Apr 29, 2024
1 parent 33a1a16 commit 4c5ad01
Show file tree
Hide file tree
Showing 6 changed files with 1,766 additions and 46 deletions.
27 changes: 0 additions & 27 deletions .github/workflows/ci.yml

This file was deleted.

9 changes: 2 additions & 7 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
ci:
autofix_commit_msg: "Chore: pre-commit autoupdate"
skip:
# pre-commit.ci cannot install WGET, so tomlint must be disabled
- tomllint

exclude: |
Expand Down Expand Up @@ -96,6 +95,7 @@ repos:
rev: 6.3.0
hooks:
- id: pydocstyle
additional_dependencies: ["tomli"]

- repo: https://github.com/PyCQA/isort
rev: 5.13.2
Expand All @@ -113,12 +113,7 @@ repos:
hooks:
- id: flake8
# Ignore all format-related checks as Black takes care of those.
args:
- --ignore=E2, W5, F401, E401
- --select=E, W, F, N
- --max-line-length=120
additional_dependencies:
- pep8-naming
args: ["--ignore=E2, W5, F401, E401", "--select=E, W, F, N", "--max-line-length=120"]

- repo: https://github.com/pre-commit/mirrors-mypy
rev: "v1.10.0"
Expand Down
26 changes: 17 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# osc-ingest-tools

python tools to assist with standardized data ingestion workflows

### Install from PyPi
Expand Down Expand Up @@ -42,10 +43,10 @@ pip install osc-ingest-tools
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 2 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 first_name 3 non-null string
1 age_in_years 3 non-null Int64
1 age_in_years 3 non-null Int64
dtypes: Int64(1), string(1)
memory usage: 179.0 bytes

Expand All @@ -55,10 +56,11 @@ memory usage: 179.0 bytes
first_name varchar,
age_in_years bigint

>>>
>>>
```

#### Adding custom type mappings to `create_table_schema_pairs`

```python
>>> df = pd.DataFrame(data, columns = ['First Name', 'Age In Years'])

Expand All @@ -68,10 +70,10 @@ memory usage: 179.0 bytes
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 2 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 first_name 3 non-null object
1 age_in_years 3 non-null int64
1 age_in_years 3 non-null int64
dtypes: int64(1), object(1)
memory usage: 176.0+ bytes

Expand All @@ -87,7 +89,7 @@ memory usage: 176.0+ bytes
### Development

Patches may be contributed via pull requests to
https://github.com/os-climate/osc-ingest-tools.
<https://github.com/os-climate/osc-ingest-tools>.

All changes must pass the automated test suite, along with various static
checks.
Expand All @@ -97,12 +99,14 @@ checks.

Enabling automatic formatting via [pre-commit](https://pre-commit.com/) is
recommended:

```
pip install black isort pre-commit
pre-commit install
```

To ensure compliance with static check tools, developers may wish to run;

```
pip install black isort
# auto-sort imports
Expand All @@ -112,6 +116,7 @@ black .
```

Code can then be tested using tox.

```
# run static checks and tests
tox
Expand All @@ -124,13 +129,16 @@ tox -e cov
```

### Releasing

To release a new version of this library, authorized developers should;

- Prepare a signed release commit updating `version` in setup.py
- Tag the commit using [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
prepended with "v"
prepended with "v"
- Push the tag

E.g.,

```
git commit -sm "Release v0.3.4"
git tag v0.3.4
Expand Down
13 changes: 11 additions & 2 deletions osc_ingest_trino/__init__.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,17 @@
from .boto3_utils import attach_s3_bucket, upload_directory_to_s3
from .dotenv_utils import load_credentials_dotenv
from .sqlcols import enforce_partition_column_order, enforce_sql_column_names, sql_compliant_name
from .sqlcols import (
enforce_partition_column_order,
enforce_sql_column_names,
sql_compliant_name,
)
from .sqltypes import create_table_schema_pairs, pandas_type_to_sql
from .trino_utils import TrinoBatchInsert, _do_sql, attach_trino_engine, fast_pandas_ingest_via_hive
from .trino_utils import (
TrinoBatchInsert,
_do_sql,
attach_trino_engine,
fast_pandas_ingest_via_hive,
)

__all__ = [
"sql_compliant_name",
Expand Down
Loading

0 comments on commit 4c5ad01

Please sign in to comment.