-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve search by adding LinkedRecords
, API docs, Kubernetes health endpoint
#23
base: master
Are you sure you want to change the base?
Conversation
MLG-1.4.0: Only support rfc6902 edit patches
ACTUM-166: Improve search by adding `LinkedRecords`; ACTUM-271: Add Swagger docs;
ACTUM-267: Add `/` endpoints for Kubernetes healthchecks
PR oipwg#19 changes
That's a rather hefty jump, can we place this option behind a config option for server operators to choose? On an initial read the changes otherwise looked pretty good to me and the Swagger docs would be very helpful. What, if any, steps would be necessary after a merge to get the Swagger docs appearing on the GH Pages for this repo? |
Modify Index Mappings
This also helps keep memory down by removing completed Multiparts from the Map. I tested this with just about one million incomplete Multiparts (with many error multiparts), and it performed admirably
I've gone in and added the config flag I have additionally found some bugs in the Multipart Indexing and have included a fix in this MR. @bitspill let me know if there are any other changes you see needing to be made |
BLUF; Improved OIP Search functionality by allowing for linked records (txids embedded a Record) to be searched as well. The
mainnet
DB size with this change increases from61GB
to193GB
, which is just over a 3x increase.Functional changes: A new field has been added to the
Artifact
index namedLinkedRecords
that stores full versions of all "Child" Records that are referenced in a "Parent" Record via a TXID. This allows for full text search of both all "Child" records fields as well as the "Parent" records fields.Changelog
Added
LinkedRecords
added to Artifact index/
endpoint to OIPd and Kibana that return a200
status (to satisfy Kubernetes healthchecks)Future recommendations
This MR only contains minimal functionality to add
LinkedRecords
ontoTenure
Artifacts that have linkedParties
orSpatialUnits
. LinkedRecords are only added on ingestion so there are the a few issues with the LinkedRecord implementation.I would recommend the following changes are eventually made.
FindTXIDs
function should be expanded to search all fields for TXID's in a recursive manneron42JsonPublishArtifact
function, it should be expanded to allow for adding a LinkedRecord when the child record is being processed byon42JsonPublishArtifact
instead of the parent record.party
is a 5 part Multipart record, and thetenure
is included in a single regular Artifact transaction. If this was the case, the parent (tenure
) gets processed before the child (party
), which causes the parent to not be able to search for/find/attach any LinkedRecords.oip042.edit.processRecord