DataStore load from local file (library + single node DAG)

[hannelita]

Add DataStore loading via API to library and build a DAG that loads a (configurable) file to (configurable) CKAN instance (either local and/or demo.ckan) using API only.

Acceptance

• Library that does loading to DataStore via API (only) load.py now has load_csv_via_api method (inserted on Single-DAG DataStore load flow using CKAN DataStore API #17), and combined with create_datastore_table does the entire flow via API
• Create DAG using datastore library api_load (leave old datastore-load-via-db-copy aside)
  • DAG can be configured with file location - ** It is possible to specify file locations for csv input and json output on the params dictionary of the DAG as shown in [Refactor][l] Splitting main lib files #18**
  • DAG can be configured with CKAN instance location (and API key) Refactor load.py to have an agnostic config #16 ([Refactor][l] Splitting main lib files #18 improves the code) allows the config of CKAN instance location (via Airflow ENV var. CKAN_SITE_URL), as well as API Key (with CKAN_SYSADMIN_API_KEY)

Tasks

• Refactor load.py to have an agnostic config
• Rewrite load_csv with API approach (similar to datapusher) Single-DAG DataStore load flow using CKAN DataStore API #17
• DAG: Specify paths and resources via kwargs or other Operators Single-DAG DataStore load flow using CKAN DataStore API #17
• Handle responses from server; i.e. collect resurce_id after create and pass it to load Single-DAG DataStore load flow using CKAN DataStore API #17
• Trigger DAG on airflow (test trigger works) [Refactor][l] Splitting main lib files #18
• Improve error messages and error handling for each DAG task. At this time they are very generic or do not reflect the real errors - Some addressed in Single-DAG DataStore load flow using CKAN DataStore API #17 and [Refactor][l] Splitting main lib files #18 ; see Create classes for Errors and Response #20 for better design porposal
• "Housekeeping" - refactor params that are being passed (see Refactor load.py to have an agnostic config #16 (comment)), refactor on method names, ensure test coverage (relates to Mocks for library #15 ), refactor and split load library files.
  Note: Tests will be moved to a separate issue; better design for passing env vars on [Refactor] Pass arguments explicitly to the function rather than pull from config #19

Analysis

Questions

[hannelita]

@rufuspollock For the DAG, we will have a choice:

1. an airflow oriented approach, which specific hooks/operators from Airflow. Pros: Fine grained control of each step, the DAG can be composed of several tasks (delete, create, load, create_indexes)
2. A generic approach using the PythonOperator only. Can be turned into a DAG of a single task (IMHO we would not make a good use of airflow) or multiple tasks (I'd need to find out how to pass the response of a task to another)

As discussed, we may go with opt 2

Both will require manipulations in the load library (see #14 + improve method responses [create will need to return the resource_id; parameterize initial JSON path]).

[hannelita]

Awaiting review of #18 to possibly close this issue

[hannelita]

FIXED with #17 and #18