Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticssearch 8 upgrade #7599

Merged
merged 28 commits into from
Feb 9, 2024
Merged

Elasticssearch 8 upgrade #7599

merged 28 commits into from
Feb 9, 2024

Conversation

josegar74
Copy link
Member

@josegar74 josegar74 commented Jan 9, 2024

This pull request updates Elasticsearch client to version 8, replacing the Elasticsearch Java High Level REST Client, that is no longer supported, with the new Elasticsearch Java API Client.

The code supports Elasticsearch servers 7.15+ and 8+

Several Sonarlint improvements have been applied.


As part of the work, two changes related to identified code to be updated have been done. These changes require to be backported to version 4.2.x:

  1. The Admin console > Tools > Clear XLink cache option has been removed, as was not implemented in GeoNetwork 4.x, and the process is executed automatically on metadata indexing to simplify the management of the XLink cache.

  2. The feature to retrieve related resources from the index, that was not used, have been removed.

Checklist

  • I have read the contribution guidelines
  • Pull request provided for main branch, backports managed with label
  • Good housekeeping of code, cleaning up comments, tests, and documentation
  • Clean commit history broken into understandable chucks, avoiding big commits with hundreds of files, cautious of reformatting and whitespace changes
  • Clean commit messages, longer verbose messages are encouraged
  • API Changes are identified in commit messages
  • Testing provided for features or enhancements using automatic tests
  • User documentation provided for new features or enhancements in manual
  • Build documentation provided for development instructions in README.md files
  • Library management using pom.xml dependency management. Update build documentation with intended library use and library tutorials or documentation

Funded by BRGM
Funded by EEA

josegar74 and others added 22 commits January 8, 2024 09:29
…RROR: Elasticsearch exited unexpectedly, with exit code 143
…n hits. A request may return no hits but can be used to check number of hits. In such case we should avoid using hits.hits.size and use hits.total.value to get number of match.
…s, clear the Xlink cache automatically before indexing and remove non-implemented code to retrieve metadata with XLink (not required anymore)
@josegar74 josegar74 added index structure change Indicate that this work introduces an index change. api change Indicate a change in the API update-release-notes PRs with this label need some notes to be added to the documentation/release notes for user action. changelog labels Jan 9, 2024
@josegar74 josegar74 added this to the 4.4.3 milestone Jan 9, 2024
@jahow
Copy link
Contributor

jahow commented Jan 25, 2024

Hi,

Thanks for the extensive work. Do you know what is the timeline for this change to be merged? And is that going to be in 4.4.3 or be part of another major version (4.6)?

Copy link

sonarcloud bot commented Feb 8, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

@jahow
Copy link
Contributor

jahow commented Feb 8, 2024

ping @josegar74 @fxprunayre for the question above 🙂 thanks!

@josegar74
Copy link
Member Author

@jahow I think the plan is to merge that in main branch. At least in our company projects we are treating 4.2.x as the stable branch, so no problem to merge this work in main for us.

I know also that probably there is no clear agreement on which branch is the stable one, but my point on that.

@jahow
Copy link
Contributor

jahow commented Feb 8, 2024

Ok so as I understand this will be part of the upcoming 4.4.3 version. Thanks for the clarification!

@fxprunayre fxprunayre merged commit fbbefc2 into main Feb 9, 2024
9 of 10 checks passed
@fxprunayre fxprunayre deleted the elasticssearch8-upgrade branch February 9, 2024 07:28
fxprunayre added a commit that referenced this pull request Feb 14, 2024
Error was Validation Failed: 1: script or doc is missing.

Related to ES8 migration #7599
@fgravin
Copy link
Member

fgravin commented Feb 14, 2024

@josegar74 on a fresh main branch, I installed ES 8 and ran ES with

   mvn install -Pes-download
   mvn exec:exec -Des-start

the index is not starting because of received plaintext http traffic on an https channel, closing connection Netty4HttpChannel

I found some stuff on stackoverflow about that but is it a known issue ?

@josegar74
Copy link
Member Author

josegar74 commented Feb 14, 2024

@fgravin I think by default ES 8 uses https, you need either to disable it or update the configuration for GeoNetwork to use https, it will require to install the certs in the JVM.

I always use an external instance using Docker with all that stuff configured.

For development, probably the best is to disable the setting xpack.security.enabled. I'll check how to manage this with using the es module to download and execute Elasticsearch.

@fgravin
Copy link
Member

fgravin commented Feb 14, 2024

Yes I've modified es/elasticsearch-8.11.3/config/elasticsearch.yml to set xpack.security.enabled: false
I think we should update the README, I will do a PR

fxprunayre added a commit that referenced this pull request Feb 14, 2024
Error was Validation Failed: 1: script or doc is missing.

Related to ES8 migration #7599
@jodygarnett
Copy link
Contributor

@davidblasby here is the PR that updated es.version.

I came here to ask that the following be updated:

  • es/README.md
  • es/docker-compose.yml

But reading above @fgravin is already aware.

@jodygarnett jodygarnett mentioned this pull request Feb 27, 2024
10 tasks
@jodygarnett
Copy link
Contributor

I do not see an issue to open with respect to this pull request, since this was merged two weeks ago I do not feel comfortable asking for a revert.

With that in mind here is a small PR updating es/ instructions #7805

fxprunayre added a commit to geonetwork/docker-geonetwork that referenced this pull request Mar 1, 2024
Main change is update to Elasticsearch 8.

Aligned with compose in main repo geonetwork/core-geonetwork#7817
Follow up geonetwork/core-geonetwork#7599.
juanluisrp pushed a commit to geonetwork/docker-geonetwork that referenced this pull request Mar 19, 2024
Main change is update to Elasticsearch 8.

Aligned with compose in main repo geonetwork/core-geonetwork#7817
Follow up geonetwork/core-geonetwork#7599.
juanluisrp pushed a commit to geonetwork/docker-geonetwork that referenced this pull request Mar 19, 2024
Main change is update to Elasticsearch 8.

Aligned with compose in main repo geonetwork/core-geonetwork#7817
Follow up geonetwork/core-geonetwork#7599.
fxprunayre added a commit that referenced this pull request Apr 9, 2024
Follow up of #7599

`result.hits().hits().size()` returns number of records in response, not total hits matching search. `numberOfRecordsMatched` is wrong.

```
  <csw:SearchResults numberOfRecordsMatched="10" numbe
```
fxprunayre added a commit that referenced this pull request Apr 10, 2024
Follow up of #7599

`result.hits().hits().size()` returns number of records in response, not total hits matching search. `numberOfRecordsMatched` is wrong.

```
  <csw:SearchResults numberOfRecordsMatched="10" numbe
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api change Indicate a change in the API changelog index structure change Indicate that this work introduces an index change. update-release-notes PRs with this label need some notes to be added to the documentation/release notes for user action.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants