Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve search by adding LinkedRecords, API docs, Kubernetes health endpoint #23

Open
wants to merge 15 commits into
base: master
Choose a base branch
from

Conversation

OstlerDev
Copy link
Collaborator

BLUF; Improved OIP Search functionality by allowing for linked records (txids embedded a Record) to be searched as well. The mainnet DB size with this change increases from 61GB to 193GB, which is just over a 3x increase.

Functional changes: A new field has been added to the Artifact index named LinkedRecords that stores full versions of all "Child" Records that are referenced in a "Parent" Record via a TXID. This allows for full text search of both all "Child" records fields as well as the "Parent" records fields.

Changelog

Added

  • Swagger document describing a few main OIPd endpoints
  • LinkedRecords added to Artifact index
  • Root / endpoint to OIPd and Kibana that return a 200 status (to satisfy Kubernetes healthchecks)

Future recommendations

This MR only contains minimal functionality to add LinkedRecords onto Tenure Artifacts that have linked Parties or SpatialUnits. LinkedRecords are only added on ingestion so there are the a few issues with the LinkedRecord implementation.

I would recommend the following changes are eventually made.

  • The FindTXIDs function should be expanded to search all fields for TXID's in a recursive manner
  • Instead of only adding LinkedRecords to the Tenure that have already been indexed (i.e. are not waiting to be completed as a multipart) in the on42JsonPublishArtifact function, it should be expanded to allow for adding a LinkedRecord when the child record is being processed by on42JsonPublishArtifact instead of the parent record.
    • i.e. Lets say the party is a 5 part Multipart record, and the tenure is included in a single regular Artifact transaction. If this was the case, the parent (tenure) gets processed before the child (party), which causes the parent to not be able to search for/find/attach any LinkedRecords.
  • Additionally, the linking of records should be expanded to occur when Artifact Edits are processed in oip042.edit.processRecord

@OstlerDev OstlerDev added the enhancement New feature or request label Jul 16, 2020
@OstlerDev OstlerDev self-assigned this Jul 16, 2020
@bitspill
Copy link
Member

The mainnet DB size with this change increases from 61GB to 193GB, which is just over a 3x increase.

That's a rather hefty jump, can we place this option behind a config option for server operators to choose?

On an initial read the changes otherwise looked pretty good to me and the Swagger docs would be very helpful.

What, if any, steps would be necessary after a merge to get the Swagger docs appearing on the GH Pages for this repo?

@OstlerDev
Copy link
Collaborator Author

I've gone in and added the config flag oip.linkedRecords.enabled and set it to false by default.

I have additionally found some bugs in the Multipart Indexing and have included a fix in this MR.

@bitspill let me know if there are any other changes you see needing to be made

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants