Duckdb

[bsantan]

• DuckDB merge implementation
• Autogeneration of merge.yaml based on input params provided
• Features use of either KGX or DuckDB for merge

[hrshdhgd]

Alright, I have my first stab at the duck db code.

I haven't tried it on the big dataset yet so no idea about the efficiency. Some points to note:

• I haven't deleted any of @bsantan 's code. Some are commented out.

• I have added unit test files with some test input and expected output. We can tweak it to cover most cases to see if the code addresses all scenarios (most at least)

• The new functions are:

  • load_into_duckdb
  • duckdb_nodes_merge
  • duckdb_edges_merge
    All the are in ducked_utils.py. In merge_kg.py I have replaced the duckdb_merge function and commented out the original code.

• Added a test file test_duckdb_utils.py that tests the code in ducked_utils.py above. We need to add more tests to test every aspect of this project. It is good practice and avoids surprises.

• There is one TODO:

# TODO: Get priority sources dynamically from the ontologies transform
    priority_sources = ["go.json", "chebi.json", "ncbitaxon_removed_subset.json"]

This can be worked on next week. There are still a lot of pieces left to fall in place.