Discussion for merging Gleaner and Nabu

Gleaner and Nabu should have a common config file, they should probably be the same app. glcon already merges them,

• option 1: just standardize configs.
• option 2: merge codebases

config files

we should use two config files, and then we can combine them into one

• services, things with urls and passwords that can be configured
• sources, implementation network

with two files, gleaner and nabu, this is already in practice glcon already uses a directory to manage the this will remove the need to 'generate' a config, though generate includes pulling from a csv file to create a sources list.

proposals

option 1:

For a common codebase, gleaner will need to import the config from nabu so all config and common stuff moves to nabu/pkg

Option 2:

We reorgianize and shorten the glcon command structure.

config

• config init (config_directory)-- initialize a config folder
• config update/generate (config_directory)-- update the sources

summon/fetch

• summon (config_directory)-- run the summon gleaner process
• mill (config_directory) -- run the mill process, using the common graph conversion code shared between gleaner and nabu

graph

• load summon|prov (--source source) (config_directory) -- was prefix
• bulkload summon|prov (--source source) (config_directory)
• release summon|prov|orgs (--source source) (config_directory) - create a release file
• object path_to_s3 (config_directory) -- upload one
• prune
• graph clear
• graph drop - drop a single named graph

utilities

• tool s3clear - clear a bucket... can we just proxy the minioadmin commands? it's a go app.
• tool identifier - read jsonld, return identifier
• tool jsonld - read jsonld, output context fixed jsonld test json ld context changes
• tool rdf - convert jsonld to rdf

** OTHERS **