API v2 support with new fields #231
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
in order to achieve support for multiple ES schemas - adds configuration for different schema locations - moves code from executables into Datura::Elasticsearch module - Datura::Options combines settings into schema path
had suffered from errors and from gem deprecation warnings
the validator ensures that all fields have either an exact mapping OR match a dynamic template, at least as far as our current simple dynamic templates go. This may need to be adjusted in the future if we start using more complex templates refactors file_type post_es method to use Elasticsearch::Index object and to rely less on repeated "returns" vs a final line at the end also adds some helpful methods like should_post? to complement should_transform? for ease
that is, previously it was assuming that nested subfields all match a top level field or dynamic mapping actually, nested fields can specify their own specific mapping that the subfields may also access....sooooooo I had to redo stuff tests should be passing! thank goodness for unit tests and tdd
reverts my earlier change, it breaks other functions
update documentation for changes to the schema
in order to achieve support for multiple ES schemas - adds configuration for different schema locations - moves code from executables into Datura::Elasticsearch module - Datura::Options combines settings into schema path
the validator ensures that all fields have either an exact mapping OR match a dynamic template, at least as far as our current simple dynamic templates go. This may need to be adjusted in the future if we start using more complex templates refactors file_type post_es method to use Elasticsearch::Index object and to rely less on repeated "returns" vs a final line at the end also adds some helpful methods like should_post? to complement should_transform? for ease
techgique
commented
Sep 14, 2024
There was a problem hiding this comment.
GitHub won't let me mark this as requiring changes because I re-made the PR myself. But please look over my review comments.
Also there are several tests failing or erroring when I run rake test from this repository.
34 runs, 76 assertions, 11 failures, 3 errors, 0 skips
We don't necessarily have to fix those on this PR. That work can be a separate PR outside of this, but I definitely want them fixed before we make a 2.0 release. That work might turn up other bugs we haven't noticed or it could just be differences between v1 and v2 API behavior.
techgique
commented
Sep 20, 2024
CHANGELOG.md
Outdated
| @@ -65,7 +61,11 @@ See new schema (2.0) documentation [here](https://github.com/CDRH/datura/docs/sc | |||
|
|
|||
| ### Migration | |||
| - check to make sure "text" xpath is doing desired behavior | |||
| - use Elasticsearch 8.5 or higher and add authentication as described above if security is enabled. See [dev docs instructions](https://github.com/CDRH/cdrh_dev_docs/blob/update_elasticsearch_documentation/publishing/2_basic_requirements.md#downloading-elasticsearch). | |||
| - use Elasticsearch 8.5 or higher and add authentication as described above if security is enabled. Add the following to `public.yml` or `private.yml` in the data repo: | |||
There was a problem hiding this comment.
Please revise the text here to reflect that one doesn't need to look above for the auth info now.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.