Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: add document nodes create at and modified at attributes #135

Merged
merged 3 commits into from
Jan 3, 2024

Conversation

ClemDoum
Copy link
Collaborator

@ClemDoum ClemDoum commented Dec 20, 2023

TODO:

PR description

Adding document nodes createdAt and modifiedAt attributes (attribute names chosen to match the follow the money schema). These attributes are read from ES document metadata.

While reading the ES metadata, we look for the DC terms created/modified metadata in priority since it's the one used in the datashare-api to define the Docuemnt.getCreationDate() (tika_metadata_dcterms_modified). In case these metadata are not found we fallback to other tika metadata.

The fallback order to creation date is :

  • tika_metadata_dcterms_created_iso8601
  • tika_metadata_creation_date_iso8601
  • tika_metadata_date_iso8601

for modification date it is:

  • tika_metadata_dcterms_modified_iso8601
  • tika_metadata_last_modified_iso8601
  • tika_metadata_modified_iso8601
  • tika_metadata_last_save_date_iso8601
  • tika_metadata_pdf_docinfo_modified_iso8601
  • tika_metadata_date_iso8601

Changes

neo4j-app

Added

  • updated the regular and admin imports to add createdAt and modifiedAt attributes to the Document nodes
  • added a indexes to Document nodes to be looked up and sorted by createdAt and modifiedAt dates

@ClemDoum ClemDoum requested a review from a team December 20, 2023 14:48
@ClemDoum ClemDoum self-assigned this Dec 20, 2023
@ClemDoum ClemDoum force-pushed the feature/add-email-creation-date branch from dacde3d to c4608f9 Compare December 20, 2023 14:59
Copy link
Contributor

@caro3801 caro3801 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. The code is well tested and the migration system is smooth, I like the fallback date attribute attention. Nice addition ! However, I'm not sure to catch why the CI is red

@ClemDoum
Copy link
Collaborator Author

Thanks for the review @caro3801

@ClemDoum
Copy link
Collaborator Author

ClemDoum commented Dec 20, 2023

Thanks for spotting the CI, actually, there was this bug #136 which made tests flaky

@ClemDoum ClemDoum force-pushed the feature/add-email-creation-date branch from c4608f9 to a933111 Compare December 20, 2023 17:13
@ClemDoum ClemDoum changed the base branch from main to fix/task-error-nack December 20, 2023 17:13
Base automatically changed from fix/task-error-nack to main December 21, 2023 08:41
@ClemDoum ClemDoum force-pushed the feature/add-email-creation-date branch from a933111 to d24ee24 Compare December 21, 2023 08:43
@ClemDoum ClemDoum force-pushed the feature/add-email-creation-date branch from d24ee24 to 7ea1753 Compare January 3, 2024 09:43
@ClemDoum ClemDoum merged commit d8211dc into main Jan 3, 2024
8 checks passed
@ClemDoum ClemDoum deleted the feature/add-email-creation-date branch January 3, 2024 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants