Elasticsearch source skeleton

[mgrauer]

@danlamanna

Can you QA this? It should give you a good start.

Also nicks the corner of #134 .

[aashish24]

👏

[mgrauer]

I wanted to do one more thing, I'm done now.

[cpatrick]

Ah memories 😸 .

[danlamanna]

@mgrauer This looks great.

We agreed Elasticsearch as a source makes sense to have in core Minerva so others can use it. If it makes sense to you I'll implement a very rudimentary backend for minerva_elasticsearch_query and get this merged in.

This will require adding elasticsearch as a Minerva dependency, which is a pretty lightweight wrapper around urllib3.

Following that, I will create analyses specific to the Elasticsearch indices people using Terra will want.

Sound good?

[mgrauer]

This all sounds good to me. I think it makes the most sense to add that additional endpoint to elasticsearch_source.py.

I think we said that endpoint will do something like

• create a Job (async local or Romanesco)
• create a Dataset, tie it into the Job (there was some discussion about adding it to the meta/minerva object in the Job, we'll have to figure out the exact specifics of this later)
• return the Job from the endpoint
• in the async Job, bring back the query result, save it as a JSON file in the Dataset
• create GeoJson from the JSON, as a collection of lat/long points, save the GeoJson file to the Dataset
• set the minerva metadata similar to this

And let me know if you have any better ideas or questions

[danlamanna]

My concern with putting that into master is that in general, Elasticsearch indices won't always contain geospatial information. I want to avoid putting anything into core that is specific to a particular domain, Memex HT data in this instance.

So if we create the general endpoint which just processes an ES query and puts the resulting JSON into a dataset, then the user can do whatever they want with it.

Then in Terra, I will create a specific analysis that takes that dataset and alters it using the code you've mentioned.

[mgrauer]

great, makes sense.

[danlamanna]

@mgrauer This is ready for review, let me know if the general approach makes sense and feel free to be as nitpicky as you'd like :)

[mgrauer]

I don't get the purpose of the elasticsearch dataset endpoint, it seems like it isn't used. Can you remove it and the test? Or else explain its use to me.

The way I see it, you have endpoints that do everything you need already, without this other endpoint

• create an elasticsearch source
• run a query job on an elasticsearch source
  • create a job
  • create a JSON dataset
  • the job runs the query on the source and uploads the result to the JSON dataset

[danlamanna]

I'll remove it, I originally added it to follow convention with the WMS endpoints.

[mgrauer]

Can you replace

if 'meta' in dataset:
...
self.model('item').setMetadata(dataset, metadata)

with

minerva_metadata = {
    'dataset_type': 'json',  
    'source_id': params['sourceId'],
    'source': 'elasticsearch',
    'elasticsearch_params': elasticsearchParams,
    'base_url': baseURL
}
updateMinervaMetadata(dataset, minerva_metadata)

?

I know this wasn't very clear.

[mgrauer]

Will you want to convert this to a romanesco job? If so, please add an issue for that.

[danlamanna]

@mgrauer Updated. This look good to you?

[mgrauer]

LGTM.