Code

jam/JDRF-pipeline - Core Pipeline#6

Code

Open

jameno wants to merge 53 commits intodevtidepool-org/donor-data-pipeline:devfrom jam/JDRF-pipelinetidepool-org/donor-data-pipeline:jam/JDRF-pipelineCopy head branch name to clipboard

+7,971-16

Commits

Commits on Jan 7, 2020

• Move old pipeline into archived folder

  jameno

  committed
  ce5fea6

  Copy full SHA for ce5fea6

Commits on Jan 21, 2020

• Add new pipeline to src

  jameno

  committed
  7d0f7e7

  Copy full SHA for 7d0f7e7

Commits on Jan 22, 2020

• Update import argument defaults from bool to string format

  jameno

  committed
  3bd726c

  Copy full SHA for 3bd726c

• Add check for userid before starting pipeline

  jameno

  committed
  a1aa601

  Copy full SHA for a1aa601

• Add an API sleep buffer argument

  Show description for 5446dfd
  jameno

  committed
  5446dfd

  Copy full SHA for 5446dfd

• Add feature to make directories if they don't exist

  Show description for a8cafbe
  jameno

  committed
  a8cafbe

  Copy full SHA for a8cafbe

• Add data dictionary and export field list files

  jameno

  committed
  d51cbad

  Copy full SHA for d51cbad

• Add try-except catch for an ignore-id-list during donor acceptance

  jameno

  committed
  618d0b6

  Copy full SHA for 618d0b6

Commits on Jan 28, 2020

• Remove original vector qualify folder creation

  jameno

  committed
  ae7475e

  Copy full SHA for ae7475e

• Remove unused library imports from split_test_train_datasets.py

  jameno

  committed
  6fcba95

  Copy full SHA for 6fcba95

• Update READMEs

  jameno

  committed
  3a9b6d5

  Copy full SHA for 3a9b6d5

• Update environment.yml

  jameno

  committed
  b75166e

  Copy full SHA for b75166e

• Update src README

  jameno

  committed
  5c0b4a3

  Copy full SHA for 5c0b4a3

Commits on Feb 5, 2020

• Add a donor distribution percentages calculation to accept_new_donors...

  Show description for 2425a67
  jameno

  committed
  2425a67

  Copy full SHA for 2425a67

Commits on Apr 16, 2020

• Convert api status errors to strings for printing

  jameno

  committed
  5a14e82

  Copy full SHA for 5a14e82

Commits on Apr 17, 2020

• Add src README section for downloading data

  jameno

  committed
  d16d34a

  Copy full SHA for d16d34a

Commits on Apr 23, 2020

• fix(getting data): Convert api response to string

  

  Jason Meno

  committed
  71f9f02

  Copy full SHA for 71f9f02

• feat(batch metadata): Skip already downloaded metadata files

  

  Jason Meno

  committed
  6637b6f

  Copy full SHA for 6637b6f

• feat(batch get data): Skip already downloaded files

  

  Jason Meno

  committed
  52b62ca

  Copy full SHA for 52b62ca

• fix(batch get data): Update metadata file userid to userID

  

  Jason Meno

  committed
  4929d84

  Copy full SHA for 4929d84

• style(saving donor list): Ignore index when saving donor list .csv

  

  Jason Meno

  committed
  cf0db95

  Copy full SHA for cf0db95

• feat(get API data): Use subprocessing and single session tokens

  

  Jason Meno

  committed
  24976e7

  Copy full SHA for 24976e7

• feat(get donor data): Add empty dataset tracking and skipping

  

  Jason Meno

  committed
  bcef9e7

  Copy full SHA for bcef9e7

• fix(data API call): Fix incorrect API error string parsing

  

  Jason Meno

  committed
  e194504

  Copy full SHA for e194504

• feat(data API call): Added wrapper and logging for failed API attempts

  

  Jason Meno

  committed
  d0dbe24

  Copy full SHA for d0dbe24

• style(donor data API): Change format of error log.

  

  Jason Meno

  committed
  adc2fc4

  Copy full SHA for adc2fc4

Commits on Apr 24, 2020

• fix(donor API): Fixed missing unary operators for printing API response

  

  Jason Meno

  committed
  a9b5ba0

  Copy full SHA for a9b5ba0

Commits on Apr 25, 2020

• refactor(env variables): Only import dotenv if an .env file is present

  Show description for 73ab37f
  

  Jason Meno

  committed
  73ab37f

  Copy full SHA for 73ab37f

• perf(data API): Improved API dataset download performance

  

  Jason Meno

  committed
  a8f41ce

  Copy full SHA for a8f41ce

• refactor(donor API): Depreciate pd.datetime call using datetime.datetime

  

  Jason Meno

  committed
  25b55c0

  Copy full SHA for 25b55c0

• Merge branch 'jam/tidepool-api-improvements' into jam/JDRF-pipeline

  

  Jason Meno

  committed
  1b76b84

  Copy full SHA for 1b76b84

• style(donor API): black auto-formatting

  

  Jason Meno

  committed
  affa999

  Copy full SHA for affa999

• fix(data API error): use argparse userid if available for error print

  

  Jason Meno

  committed
  b6aa985

  Copy full SHA for b6aa985

• Merge branch 'jam/tidepool-api-improvements' into jam/JDRF-pipeline

  

  Jason Meno

  committed
  1e3ffc4

  Copy full SHA for 1e3ffc4

• fix(API failure writing): Change userid to data_args userid

  

  Jason Meno

  committed
  aa49ace

  Copy full SHA for aa49ace

• perf(data API): Reduce data API chunk size to 90 days from 182 days

  

  Jason Meno

  committed
  85bfd42

  Copy full SHA for 85bfd42

• docs(donor API): Update script header and argparse information

  

  Jason Meno

  committed
  a9a8d7c

  Copy full SHA for a9a8d7c

• docs(donor API): Added docstring to main get data function

  

  Jason Meno

  committed
  7d3bff0

  Copy full SHA for 7d3bff0

Commits on Jun 25, 2020

• perf(get device data): Switch to orjson library for performance

  

  Jason Meno

  committed
  24ec758

  Copy full SHA for 24ec758

• feat(environment): Add dependencies

  

  Jason Meno

  committed
  56fe740

  Copy full SHA for 56fe740

• style(get data): Cleanup formatting of batch-get-donor-data

  

  Jason Meno

  committed
  8e75348

  Copy full SHA for 8e75348

• perf(get metadata): Improve metadata batch processing login performance

  

  Jason Meno

  committed
  d146a3f

  Copy full SHA for d146a3f

• feat: limit default metadata and device data batch accounts downloaded

  

  Jason Meno

  committed
  5bb7d95

  Copy full SHA for 5bb7d95

Commits on Jul 23, 2020

• feat: Add donor notes processing scripts

  

  Jason Meno

  committed
  8587f7f

  Copy full SHA for 8587f7f

• Merge pull request #7 from tidepool-org/jam/tidepool-api-improvements

  Show description for 03916e5
  

  Jason Meno

  authored
  03916e5

  Copy full SHA for 03916e5

Commits on Dec 14, 2020

• Update README.md

  

  Jason Meno

  authored
  55eba05

  Copy full SHA for 55eba05

• Update README.md

  

  Jason Meno

  authored
  843a855

  Copy full SHA for 843a855

• feat: Remove default metadata file capture limit

  

  Jason Meno

  authored
  623f86d

  Copy full SHA for 623f86d

• feat: Remove default limit of 3 dataset downloads

  

  Jason Meno

  authored
  b252058

  Copy full SHA for b252058

• feat: Remove check for uploads from pipeline

  

  Jason Meno

  authored
  e021fe6

  Copy full SHA for e021fe6

• feat: Remove batch data download limit

  

  Jason Meno

  authored
  c1783fd

  Copy full SHA for c1783fd

• feat: change data batch download size to 14 days

  

  Jason Meno

  authored
  44f983d

  Copy full SHA for 44f983d

Commits on Dec 15, 2020

• refactor: Update multiple pipeline components

  jameno

  committed
  6766af2

  Copy full SHA for 6766af2