Code

Record linkage for SEC to EIA#120

Code

Open

katie-lamb wants to merge 171 commits intomaincatalyst-cooperative/mozilla-sec-eia:mainfrom splink-skeletoncatalyst-cooperative/mozilla-sec-eia:splink-skeletonCopy head branch name to clipboard

+14,101-11

Commits

Commits on Aug 13, 2024

• Initial dagster integration

  zschira

  committed
  5767035

Commits on Aug 14, 2024

• Update validate integration test to dagster infra

  zschira

  committed
  9d9fbfd

Commits on Aug 20, 2024

• Merge branch 'main' into dagster_integration

  zschira

  committed
  3da9659

Commits on Aug 27, 2024

• Generalize mltools

  zschira

  committed
  ee77e7a

Commits on Aug 28, 2024

• Reorg repo to move towards generalized modelling repo

  zschira

  committed
  53d3354

• Change library module structure

  zschira

  committed
  014bcb1

• Create turn experiment_tracking into sub-package

  zschira

  committed
  5404148

• Remove unused function

  zschira

  committed
  886614f

• Gracefully handle mlflow run on failure

  zschira

  committed
  dec80b8

• Fix variable name

  zschira

  committed
  e725f3d

• Change experiment tracker resource names

  zschira

  committed
  df44ed5

• Add mlflow artifact io-manager

  zschira

  committed
  93da052

Commits on Aug 29, 2024

• Simplify pudl_models decorator

  zschira

  committed
  07713e9

• Split extraction logging into two funcs

  zschira

  committed
  5d89ec6

• Add mlflow metrics io-manager

  zschira

  committed
  c57818a

• Change pudl_model to pudl_pipeline

  zschira

  committed
  625783b

Commits on Aug 30, 2024

• Add validation pipeline

  zschira

  committed
  4f50a7b

Commits on Sep 2, 2024

• Streamline construction of dagster jobs for running/testing pudl models

  zschira

  committed
  f6ab22c

• Remove old comment

  zschira

  committed
  f20fb7d

Commits on Sep 3, 2024

• Add ex21 to dagster jobs

  zschira

  committed
  92e2e00

• Prep for multiple code locations

  zschira

  committed
  520e6d1

• Add top-level worksapce file

  zschira

  committed
  e99ee1a

• Restructure docs

  zschira

  committed
  559c0e6

• Add train model job

  zschira

  committed
  93d02f3

• Log mlflow artifacts as parquet until csv is fixed

  zschira

  committed
  5190bf9

Commits on Sep 4, 2024

• Fix ex21 extraction

  zschira

  committed
  ca9599e

• Add development section to docs

  zschira

  committed
  7e7a503

• Fix integration tests

  zschira

  committed
  61f48c3

• Don't run ruff on notebooks

  zschira

  committed
  0fd8ffc

• xfail ex21 integration test

  zschira

  committed
  97d5587

Commits on Sep 5, 2024

• Add parquet upath io-manager

  zschira

  committed
  ace268b

• Remove nb-output clear

  zschira

  committed
  fb1feeb

• Test docker deployment

  zschira

  committed
  294ec72

• Chunk ex 21 extraction

  zschira

  committed
  4de51b3

Commits on Sep 6, 2024

• Fix asign copy

  zschira

  committed
  214e28f

• Add job for testing ex21 resource usage

  zschira

  committed
  c5736e0

• Merge branch 'test_parquet_logging' into dagster_integration

  zschira

  committed
  4a81e88

• Remove test docker files

  zschira

  committed
  ec39633

• Remove complex asset factory

  zschira

  committed
  101ccf1

• Parallelize ex21 extraction

  zschira

  committed
  7e0c5a5

• Don't chunk in inference module

  zschira

  committed
  080d790

• Handle failures in converting to pdf

  zschira

  committed
  44dfc52

• Delete cached pdfs early

  zschira

  committed
  6e24157

Commits on Sep 9, 2024

• Add metadata to chunk_filings

  zschira

  committed
  cd06d07

• Catch oom errors while extracting ex21

  zschira

  committed
  e3e8c45

• Fix ex21 gcs io-manager

  zschira

  committed
  350defb

• Fix partitions for basic 10k extraction.

  zschira

  committed
  3c80b72

• Cache layoutlm locally

  zschira

  committed
  31971b7

• Fix caching model

  zschira

  committed
  634a050

• Remove bad call

  zschira

  committed
  69ee4c0

Commits on Sep 10, 2024

• Test own_per conversion

  zschira

  committed
  63d6600

• Add pandera types for output tables

  zschira

  committed
  c8490d4

• Add missing entities module

  zschira

  committed
  fa4f57d

• Don't cache model, load with io manager

  zschira

  committed
  35e917d

• Remove float conversion

  zschira

  committed
  a7b1c7f

• Add hypothesis to deps

  zschira

  committed
  f019117

• Make own_per str

  zschira

  committed
  d7d13d8

• Remove astype

  zschira

  committed
  70f5293

• Validate ex21 return types

  zschira

  committed
  e406092

Commits on Sep 11, 2024

• Clean model download temp dir

  zschira

  committed
  f3835d9

• Fix model return type

  zschira

  committed
  3c995cd

• Catch errors in creating ex 21 dataset

  zschira

  committed
  ef55e4b

• Fix column name

  zschira

  committed
  b37450a

Commits on Sep 12, 2024

• Try to catch empty pdf errors

  zschira

  committed
  06b18ed

• Print traceback in caught exception

  zschira

  committed
  abfc006

• Fix empty pdf check

  zschira

  committed
  ff92a55

• Actually fix empty pdf check?

  zschira

  committed
  8aa8c95

Commits on Sep 18, 2024

• Use UPath in GCSArchive

  zschira

  committed
  43600bc

• Make _configure_mlflow a standalone function

  zschira

  committed
  05ad82c

• Merge branch 'main' into error_handling_improvements

  zschira

  committed
  fddc3b2

• Try to skip notebooks in ruff check

  zschira

  committed
  99fc7ed

Commits on Sep 19, 2024

• Pull integration test fixes from main

  zschira

  committed
  b135500

• Fix typos in README.rst

  zschira

  authored
  6e868f2

• Cache downloaded layoutlm in dagster home

  zschira

  committed
  df4fd09

• Merge branch 'error_handling_improvements' of github.com:catalyst-cooperative/mozilla-sec-eia into error_handling_improvements

  zschira

  committed
  74d237d

• Fix broken test

  zschira

  committed
  3642765

Commits on Sep 20, 2024

• fix rename filings

  katie-lamb

  committed
  830bd74

• fix paths to cache training data

  katie-lamb

  committed
  2cd1fe6

• update root dir path

  katie-lamb

  committed
  64dc8c5

• Fix UPath initialization

  zschira

  committed
  226d91c

• Fix path in test

  zschira

  committed
  3c17d33

• Create huggingface dataset outside model execution

  zschira

  committed
  df69f42

• small fixes to path handling

  katie-lamb

  committed
  2d3345c

• Merge branch 'error_handling_improvements' into second-pass-ex21-improvements

  katie-lamb

  committed
  46e7b40

Commits on Sep 23, 2024

• Minor fixes

  zschira

  committed
  6f9d34a

• Start migrating model training to notebook

  zschira

  committed
  07d500a

Commits on Sep 24, 2024

• Create dataset as dataframe for logging

  zschira

  committed
  81813a7

• Modify dataset return type

  zschira

  committed
  5174ed7

• Fix dataset types for model signature

  zschira

  committed
  7a572c0

Commits on Sep 25, 2024

• Migrate ex 21 model training to a notebook

  zschira

  committed
  5728026

Commits on Oct 3, 2024

• Merge initial notebook migration (broken)

  zschira

  committed
  5fbbfff

Commits on Oct 4, 2024

• Split dataset loading into separate assets

  zschira

  committed
  37edd50

• Minor notebook fixes

  zschira

  committed
  d6889e3

• Fix import in notebook

  zschira

  committed
  d5e013a

• add device to pipeline

  zschira

  committed
  f9810db

• Fix signature inference

  zschira

  committed
  2760881

• Fix notebook dagster config

  zschira

  committed
  1dcacfa

• Fix config param name

  zschira

  committed
  39bb45b

Commits on Oct 5, 2024

• Partition training data

  zschira

  committed
  cb83862

• Add partitions to notebook asset

  zschira

  committed
  c71593c

Commits on Oct 6, 2024

• Update ex21 labels

  zschira

  committed
  4efa515

• Use run name for specifying training runs

  zschira

  committed
  581b2e3

• Rework how notebook is configured

  zschira

  committed
  c67a1be

• Finetune configuration

  zschira

  committed
  b8a5b24

Commits on Oct 7, 2024

• separate inference dataset creation from model prediction

  zschira

  committed
  45d5cf8

• Remove deprecated inference module

  zschira

  committed
  3e15b1f

Commits on Oct 8, 2024

• Add notebook for training ex21 classifier

  zschira

  committed
  60a1260

• Pull in model updates

  zschira

  committed
  4105110

• Update classifier model

  zschira

  committed
  4d29037

Commits on Oct 9, 2024

• Fix set on copy pandas issue

  zschira

  committed
  85c44ff

• Fix model uri's

  zschira

  committed
  52e3580

• Fix indices in extraction model

  zschira

  committed
  b709053

• Fix typo

  zschira

  committed
  b8dad3c

Commits on Oct 10, 2024

• Add asset factory for loading models

  zschira

  committed
  e6b29ff

• Catch layout classification NaN exception

  zschira

  committed
  3d11777

• Use GCS pickle io-manager

  zschira

  committed
  df5fe0d

Commits on Oct 11, 2024

• Switch gcs pickle io manager to upath based

  zschira

  committed
  d6c41a2

• Remove duplicate logger

  zschira

  committed
  ddd2263

• Fix config warnings

  zschira

  committed
  93bffcb

• Test pin sphinx

  zschira

  committed
  d717caa

Commits on Oct 13, 2024

• add splink and model to environment

  katie-lamb

  committed
  09cd189

Commits on Oct 14, 2024

• Catch errors while normalizing bounding boxes

  zschira

  committed
  15be127

• Fix call to pandera example

  zschira

  committed
  4117d0a

• Fix handle failures in converting to pdf

  zschira

  committed
  8c8dd60

• Actually fix handle failures in converting to pdf

  zschira

  committed
  ff821b5

Commits on Oct 16, 2024

• Add model documentation to sec10k readme

  zschira

  committed
  a8eb359

• Fix ex 21 validation integration test

  zschira

  committed
  dc160ac

• Improve classifier error handling

  zschira

  committed
  10b24a9

• Fully broaden classifier errors

  zschira

  committed
  ad54979

• add more docs on running the notebooks

  zschira

  committed
  672e123

Commits on Oct 21, 2024

• add splink notebooks and preprocessing functions

  katie-lamb

  committed
  b95b2fb

Commits on Oct 22, 2024

• clean up feature creation in paragraph classifier

  katie-lamb

  committed
  2dbdcaa

• fix feature creation function

  katie-lamb

  committed
  cda3225

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 509b7a0
  pre-commit-ci[bot]

  committed
  509b7a0

• small fixes to read in comments in tracking dataframe

  katie-lamb

  committed
  8855e5e

• Merge branch 'prep_paragraph_classifier' of https://github.com/catalyst-cooperative/mozilla-sec-eia into prep_paragraph_classifier

  katie-lamb

  committed
  f806ad8

Commits on Oct 23, 2024

• updates to model pipeline

  katie-lamb

  committed
  590ba60

• take out logging messages

  katie-lamb

  committed
  3db47d4

• Merge branch 'prep_paragraph_classifier' into splink-skeleton

  katie-lamb

  committed
  858ecff

• make pudl editable

  katie-lamb

  committed
  61c8abf

Commits on Nov 27, 2024

• add in record linkage modules

  katie-lamb

  committed
  e5148d8

Commits on Nov 29, 2024

• fix errors with asset creation

  katie-lamb

  committed
  88f17f2

Commits on Nov 30, 2024

• clean up sec output table creation

  katie-lamb

  committed
  c9b62ba

• splink notebook change

  katie-lamb

  committed
  01b2d23

• Merge branch 'main' into splink-skeleton

  katie-lamb

  committed
  5d9419b

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 30d22c9
  pre-commit-ci[bot]

  committed
  30d22c9

• fix pre commit

  katie-lamb

  committed
  3fca3c9

• Merge branch 'splink-skeleton' of https://github.com/catalyst-cooperative/mozilla-sec-eia into splink-skeleton

  katie-lamb

  committed
  a50dd9d

Commits on Dec 1, 2024

• update python dependency in test environment

  katie-lamb

  committed
  d53ab25

• update github tox env

  katie-lamb

  committed
  2eb1555

Commits on Dec 2, 2024

• restructure input table assets

  katie-lamb

  committed
  44eb70b

• include pseudo code of SEC output table module

  katie-lamb

  committed
  7563df8

Commits on Dec 3, 2024

• Try using conda env to run tox

  jdangerx

  committed
  3c88ff2

• Add PUDL dependency and restrict to Py3.12

  jdangerx

  committed
  fb3d772

• Guess you don't need to specify tox env with setup-micromamba

  jdangerx

  committed
  390770f

• Install GDAL version via conda since we rely on PUDL now

  jdangerx

  committed
  8e8beee

• notebook has cells for SEC and EIA hook up

  katie-lamb

  committed
  50e7b7e

Commits on Dec 4, 2024

• Merge pull request #123 from catalyst-cooperative/splink-skeleton-tox-fix

  Show description for 224721b
  jdangerx

  authored
  224721b

• Fix dagster setup for record linkage inputs

  zschira

  committed
  7dc78e1

Commits on Dec 9, 2024

• fix util functions

  katie-lamb

  committed
  daa8f0a

Commits on Dec 10, 2024

• Handle missing partitions in extracted data

  zschira

  committed
  dbefe34

Commits on Dec 11, 2024

• Fix basic_10k partitions

  zschira

  committed
  97f5d68

• debug materialization of rl input assets

  katie-lamb

  committed
  b26f1f8

Commits on Dec 16, 2024

• clean up notebook to work with dagster assets

  katie-lamb

  committed
  acaf3d1

Commits on Dec 17, 2024

• clean up new structure of sec assets

  katie-lamb

  committed
  f4cceb7

Commits on Dec 18, 2024

• add in final match between ex 21 subs and eia utilities

  katie-lamb

  committed
  fa9e52e

• remove sec output table module

  katie-lamb

  committed
  599ae87

• add drop duplicates on sec company id

  katie-lamb

  committed
  c340718

Commits on Dec 19, 2024

• clean up notbook

  katie-lamb

  committed
  24de7d6

• add markdown cell note

  katie-lamb

  committed
  26b1a72

• make asset not multi asset

  katie-lamb

  committed
  70427a0