Ignoring /run and /var/run when building directories into RPM packages. #285

…dexShardAllocation test (opensearch-project#11878) (opensearch-project#12091) (cherry picked from commit f9ab801) Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

… Enable trace logs on replication and recovery (opensearch-project#12088) (opensearch-project#12090) (cherry picked from commit 64c00b3) Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…t#8394) (opensearch-project#12080) * fixed conflicts Signed-off-by: fahadshamiinsta <[email protected]> * applying spotless Java check Signed-off-by: fahadshamiinsta <[email protected]> * added a comment to test helper method Signed-off-by: fahadshamiinsta <[email protected]> * added a comment to GoogleApplicationDefaultCredentials class with document reference Signed-off-by: fahadshamiinsta <[email protected]> * to rerun gradle checks Signed-off-by: fahadshamiinsta <[email protected]> * increasing coverage by adding another test Signed-off-by: fahadshamiinsta <[email protected]> * test name change Signed-off-by: fahadshamiinsta <[email protected]> * rerun ci Signed-off-by: fahadshamiinsta <[email protected]> * rerun ci Signed-off-by: fahadshamiinsta <[email protected]> * force push to rerun ci Signed-off-by: fahadshamiinsta <[email protected]> * pushing to trigger ci checks Signed-off-by: fahadshamiinsta <[email protected]> --------- Signed-off-by: fahadshamiinsta <[email protected]> (cherry picked from commit 4c283a7) Co-authored-by: Fahad Shami <[email protected]>

…rch-project#12098) (cherry picked from commit 7526f86) Signed-off-by: Thomas Seidl <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…rch-project#12070) (opensearch-project#12100) (cherry picked from commit 2f8d267) Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pensearch-project#12107) Signed-off-by: Prabhat Sharma <[email protected]> Co-authored-by: Prabhat Sharma <[email protected]> (cherry picked from commit 829215c)

) (opensearch-project#12078) * Updating Ip fields to use doc_values to search * Fix IP tests * Fix skip to allow yaml test to pass on main * Update tests to use existing test file * Changing skip version to match bwc * Using exact match instead of range * Spotless * Fix IP field tests * Fix spotless + precommit failure * Get point out of query and into value * Fix term tests * Add skip test logic to only doc_values test --------- (cherry picked from commit cc9ee9d) Signed-off-by: Harsha Vamsi Kalluri <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…Start (opensearch-project#12094) (opensearch-project#12094) Signed-off-by: David Zane <[email protected]> (cherry picked from commit fb2c5f2)

(cherry picked from commit cfcb128) Signed-off-by: Sarthak Aggarwal <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…#12079) (opensearch-project#12115) * Adding logging for AzureStats * Debugging * Removing debugging lines * Addressing comments * Addressing feedback 2 --------- (cherry picked from commit 16c5257) Signed-off-by: Sarat Vemulapalli <[email protected]> Signed-off-by: Sarat Vemulapalli <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ct#12093) (opensearch-project#12140) (cherry picked from commit 90a815e) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…imization to kick in (opensearch-project#12089) (opensearch-project#12129) * Add advance(int) for numeric values in order to allow point based optimization to kick in Signed-off-by: Andriy Redko <[email protected]> * Address code review comments Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 4471a8d)

…ensearch-project#12074) (opensearch-project#12152) Signed-off-by: Jay Deng <[email protected]>

…ats until sync to remote completes (opensearch-project#11896) (opensearch-project#12143) (cherry picked from commit 57cc0dd) Signed-off-by: bansvaru <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…very (opensearch-project#11720) (opensearch-project#12161) * Giving time for snapshot recovery/local time to upload all the data to remote (cherry picked from commit 4d055b8) Signed-off-by: Gaurav Bafna <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

… (opensearch-project#11803) (opensearch-project#11902) (cherry picked from commit 517f091) Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…2164) (cherry picked from commit 985c411) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ch-project#12139) (opensearch-project#12145) The test currently blocks on receiving a response to the request for the "internal:testNotSeen" action. However, that response is sent from TransportService before the trace logger [writes its log message][1]. Since the test was not polling for this "sent response" log message to appear that meant it was possible for the test to remove/stop the mock log appender concurrently with the logging of that final message. The fix is to include this final log message as an expectation, so the test will poll until this message appears and the logger should be quiescent when the appender is removed and stopped. [1]: https://github.com/opensearch-project/OpenSearch/blob/71f1fabe149fd0777edf44502ace4a8f0911feeb/server/src/main/java/org/opensearch/transport/TransportService.java#L1273 (cherry picked from commit f55b9e0) Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ribution/packages (opensearch-project#12167) (opensearch-project#12169) * Bump com.netflix.nebula.ospackage-base in /distribution/packages Bumps com.netflix.nebula.ospackage-base from 11.6.0 to 11.8.0. --- updated-dependencies: - dependency-name: com.netflix.nebula.ospackage-base dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit aad2630) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…-project#11027) (opensearch-project#12171) * Enable Fuzzy codec for doc id fields using a bloom filter (cherry picked from commit 0a88963) Signed-off-by: mgodwan <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

) (opensearch-project#12172) * Bump org.gradle.test-retry from 1.5.4 to 1.5.8 Bumps org.gradle.test-retry from 1.5.4 to 1.5.8. --- updated-dependencies: - dependency-name: org.gradle.test-retry dependency-type: direct:production update-type: version-update:semver-patch ... * Update changelog --------- (cherry picked from commit 60b2ac4) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…arch-project#11971) * Added javadoc for new files/packages * Added changelog * Fixing javadoc warnings * Addressing comments * Addressing additional minor comments * Moving non null check to builder for OS onHeapCache * Adding package-info for new packages * Removing service and adding different cache interfaces along with event listener support * Fixing gradle missingDoc issue * Changing listener logic, removing tiered cache integration with IRC * Adding opensearch.internal tag for LoadAwareCacheLoader * Fixing thread safety issue * Remove compute function and event listener logic change for TieredCache * Making Cache.compute function private * Adding javadoc and more test for cache.put * Adding write locks to refresh API as well * Removing unwanted EventType class and refactoring one UT * Removing TieredCache interface --------- (cherry picked from commit ebda963) Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ect#12062) (opensearch-project#12175) * [Metric Framework] Adds support for Histogram metric * Adds test * Addresses review comments * Adds change log * Fixed spotless * Fixes javadoc * Fixes javadoc * Fixes test * Removes explicit approach * Removes explicit approach * Addresses review comments --------- (cherry picked from commit a4bc4af) Signed-off-by: Gagan Juneja <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Gagan Juneja <[email protected]>

…project#12133) * Added spotless for json files * Excluded json files with new line tests --------- (cherry picked from commit 3c07461) Signed-off-by: Owais Kazi <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ntroller (opensearch-project#12111) * Added changes for AdmissionControl Interceptor and AdmissionControlService for RateLimiting (opensearch-project#9286) * Changes for AdmissionControl Interceptor and AdmissionControlService for RateLimiting (opensearch-project#9286) Signed-off-by: Ajay Kumar Movva <[email protected]> * Integrate CPU AC with ResourceUsageCollector and add CPU AC stats to nodes/stats (opensearch-project#10887) * Added changes to integrade cpu AC to ResourceUsageCollector and Emit Stats Signed-off-by: Ajay Kumar Movva <[email protected]> Co-authored-by: Bharathwaj G <[email protected]> * Updating Version to 2.12 for the AC Stats Signed-off-by: Ajay Kumar Movva <[email protected]> --------- Signed-off-by: Ajay Kumar Movva <[email protected]> Co-authored-by: Bharathwaj G <[email protected]> Co-authored-by: Ajay Kumar Movva <[email protected]>

…ect#12183) (opensearch-project#12188) (cherry picked from commit cf3bbb8) Signed-off-by: mgodwan <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ener (opensearch-project#11526) (opensearch-project#12192) * Support dynamically adding SearchRequestOperationsListener (opensearch-project#11526) Along the way, also refactored TransportSearchAction.TimeProvider, so that it's no longer a (redundant) listener. --------- Signed-off-by: Chenyang Ji <[email protected]> (cherry picked from commit 6aab360) * fix compilation error Signed-off-by: Chenyang Ji <[email protected]> --------- Signed-off-by: Chenyang Ji <[email protected]>

* Force version of logback-core and logback-classic to 1.2.13 (opensearch-project#11521) * force version of logback-core and logback-classic to 1.2.13 Signed-off-by: Marc Handalian <[email protected]> * add changelog Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> * Bump jetty version in hdfs-fixture to 9.4.53.v20231009 (opensearch-project#11539) * Bump jetty version in hdfs-fixture to 9.4.53.v20231009 Signed-off-by: Marc Handalian <[email protected]> * fix changelog Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> * Exclude apache avro version included with hadoop-minicluster (opensearch-project#11564) Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> Co-authored-by: Marc Handalian <[email protected]>

…search-project#12199) * Query insights plugin implementation Signed-off-by: Chenyang Ji <[email protected]> * Increase JavaDoc coverage and update PR based comments Signed-off-by: Chenyang Ji <[email protected]> * Refactor record and service to make them generic Signed-off-by: Chenyang Ji <[email protected]> * refactor service for improving multithreading efficiency Signed-off-by: Chenyang Ji <[email protected]> --------- Signed-off-by: Chenyang Ji <[email protected]> (cherry picked from commit 3cbf54e)

…ode (opensearch-project#9795) (opensearch-project#12193) Signed-off-by: Harish Bhakuni <[email protected]> Co-authored-by: Harish Bhakuni <[email protected]>

…oject#12203)

…ect#12196) (opensearch-project#12206) Signed-off-by: Jay Deng <[email protected]>

…oject#12200) (opensearch-project#12210) (cherry picked from commit 13dc342) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

(cherry picked from commit 2ce15a5) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ensearch-project#12216) * Add remove_by_pattern ingest processor * Modify change log * Remove some duplicated checks * Add more yml test case * Fix typo --------- Signed-off-by: Gao Binlong <[email protected]> (cherry picked from commit 8d54278)

…roject#10136) (opensearch-project#12220) * Added custom sampler support based on action in request * UT Fix * Added Transport action sampler, which will sample based on different probability for all actions * Added Transport action sampler, which will sample based on different probability for all actions. Also added setting to define order of samplers * Added missing java-doc * Moving sampler class settings to OtelTelemetry setting * Minor refactor * Refactored to use chain of samplers * Addressed comments * Addressed comments to move action_probability to OtelTelemetrySettings * Updated eror msg returned when Sampler class is not found * Added UT for OTelSamplerFactory * minor refactor * minor refactor * spotless check * Updating OtelTelemetryPlugin.get() method * Addressed comments * minor refactor * addressed comments * Updated transport action sampler * Empty-Commit * Empty-Commit --------- (cherry picked from commit 445bf1f) Signed-off-by: Dev Agarwal <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ests (opensearch-project#12157) (opensearch-project#12224) Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>

…search-project#12005) (opensearch-project#12221) * bug fix to have RestActions rawPath as operation names, query params as tags and avoid using exact URI (opensearch-project#10791) Signed-off-by: Atharva Sharma <[email protected]> * Added UTs Signed-off-by: Atharva Sharma <[email protected]> * Added parameterized tests Signed-off-by: Atharva Sharma <[email protected]> * improvised parameterized tests using annotations Signed-off-by: Atharva Sharma <[email protected]> * removed unused annotations Signed-off-by: Atharva Sharma <[email protected]> * Added details in CHANGELOG.md Signed-off-by: Atharva Sharma <[email protected]> * Addressed spotlessJavaCheck failures Signed-off-by: Atharva Sharma <[email protected]> --------- Signed-off-by: Atharva Sharma <[email protected]> Signed-off-by: Atharva Sharma <[email protected]> Co-authored-by: Atharva Sharma <[email protected]> (cherry picked from commit c0fca74)

…t#11954) (opensearch-project#12212) (cherry picked from commit 0d50525) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Signed-off-by: Andriy Redko <[email protected]>

…ch-project#11505) (opensearch-project#11914) We can do efficient DateHistogram aggregation under composite aggregations. --------- (cherry picked from commit d5a6f99) Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…nsearch-project#10275) (opensearch-project#11972) --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]> Co-authored-by: Kiran Prakash <[email protected]>

…12121) (opensearch-project#12244) * Add community_id ingest processor (opensearch-project#12121) * Add community id ingest processor Signed-off-by: Gao Binlong <[email protected]> * Modify change log Signed-off-by: Gao Binlong <[email protected]> * Optimize the code Signed-off-by: Gao Binlong <[email protected]> --------- Signed-off-by: Gao Binlong <[email protected]> (cherry picked from commit a0b5198) * Modify change log Signed-off-by: Gao Binlong <[email protected]> --------- Signed-off-by: Gao Binlong <[email protected]>

…r to use assertbusy (opensearch-project#11981) (opensearch-project#12109) * fix flaky test - handle IllegalStateException and throw AssertionError to use assertbusy * Empty Commit to retry the build --------- (cherry picked from commit bf5e628) Signed-off-by: bansvaru <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

) (opensearch-project#12136) (cherry picked from commit f715ee1) Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ver range (opensearch-project#11441) (opensearch-project#12271) * Add support for dependencies in plugin descriptor properties with semver range (opensearch-project#1707) * Remove unused gson licenses * Maintain bwc in PluginInfo with addition of semver range * Added support for list of ranges * Add bwc tests and restrict range list size to 1 * Update SemverRange javadoc * Minor change to trigger jenkins re-run * Use jackson instead of gson * Remove jackson databind and annotations dependency from server * nit fixes * Minor change to re-run jenkins workflow --------- Signed-off-by: Abhilasha Seth <[email protected]>

…ject#12269) (cherry picked from commit 7b99274) Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…-project#12265) (cherry picked from commit 4d044e4) Signed-off-by: Peter Nied <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…level (opensearch-project#12279) * Apply fast date histogram optimization at the segment level (opensearch-project#12073) --------- Signed-off-by: bowenlan-amzn <[email protected]> (cherry picked from commit 9a0a69f) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Fix: reset the filter built at segment level for date histogram optimization (opensearch-project#12267) --------- Signed-off-by: bowenlan-amzn <[email protected]> --------- Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…roject#9580 (opensearch-project#11918) (opensearch-project#12275) (cherry picked from commit c6cebc7) Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Signed-off-by: Mikhail Khludnev <[email protected]>

Signed-off-by: Andrew Ross <[email protected]>

* Add 2.12 release notes to 2.x branch Signed-off-by: Andrew Ross <[email protected]> * Remove unreleased PR from release notes Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: Andrew Ross <[email protected]>

…earch-project#12301) (opensearch-project#12305) Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit bff8eb7)

…s-fixture (opensearch-project#12290) (opensearch-project#12316) * Bump com.squareup.okio:okio in /test/fixtures/hdfs-fixture Bumps [com.squareup.okio:okio](https://github.com/square/okio) from 3.7.0 to 3.8.0. - [Changelog](https://github.com/square/okio/blob/master/CHANGELOG.md) - [Commits](square/okio@parent-3.7.0...parent-3.8.0) --- updated-dependencies: - dependency-name: com.squareup.okio:okio dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 84750b3) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…Request classes. (opensearch-project#12314) fix opensearch-project#12144 Signed-off-by: Guillaume Alvarez <[email protected]>

…endencies from jdk15to18 to jdk18on (opensearch-project#12326) Signed-off-by: Stephen Crawford <[email protected]>

@ignore

…sts (opensearch-project#11773) (opensearch-project#12270) * Run few tests with Segment Replication enabled. * Update reason for ignoring test. * remove @ignore to resolve :server:forbiddenApisInternalClusterTest check. * fix spotlessCheck. * add conditional logic of force refresh. * Address comments on PR. * fix failing errors. * Use parameterization for running segment replication tests. * Fix failing tests. * Fix failing test. * add new waitForReplication() and refactor. * Address comments on PR and revert back changes made to SegmentReplication Tests. * revert changes made to Segrep tests. * Refactor and address comments. * fix failure of using forbidden api new Random(). * Add comments to debug. * Remove non-critical tests from running with segrep. * Fix test to run with segrep. * separate out refresh and waitForReplication into different methods. * refactor with usage of waitForReplication(). * fix parameters passed in factory for IndexStatsIT. * Update IndexstatsIT to run with segrep --------- (cherry picked from commit 30c4210) Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…t#10551) (opensearch-project#12329)

…pensearch-project#12340) (cherry picked from commit 03e415e) Signed-off-by: bowenlan-amzn <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…) (opensearch-project#12335) If you configure IntelliJ to use JDK 21 then all unit tests will fail to run in the IDE when attempting to set the security manager in the test bootstrap code. This change configures all tests to include the required VM argument to enable the security manager. (cherry picked from commit f73c82a) Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

opensearch-project#12179) (opensearch-project#12343) * Fix a flaky test that relies on terminate_after for an exact count match. (Issue # 10435) * Make comments HTML Javadoc friendly. * Fix javadoc syntax --------- (cherry picked from commit 8fe15e6) Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Andrew Ross <[email protected]>

…t#12348) (cherry picked from commit 4b51a85) Signed-off-by: RS146BIJAY <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…gging is enabled (opensearch-project#12328) (opensearch-project#12347)

…tart() will always be followed by onPhaseEnd() within AbstractSearchAsyncAction (opensearch-project#12293) (opensearch-project#12349) * Add support of special WrappingSearchAsyncActionPhase so the onPhaseStart() will always be followed by onPhaseEnd() within AbstractSearchAsyncAction Signed-off-by: Andriy Redko <[email protected]> * Address code review comments Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit b19e427)

…#12367) (opensearch-project#12370) * Bump gradle/wrapper-validation-action from 1 to 2 Bumps [gradle/wrapper-validation-action](https://github.com/gradle/wrapper-validation-action) from 1 to 2. - [Release notes](https://github.com/gradle/wrapper-validation-action/releases) - [Commits](gradle/wrapper-validation-action@v1...v2) --- updated-dependencies: - dependency-name: gradle/wrapper-validation-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 75d64e2) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ter with "plain… (opensearch-project#12357) * Fix the "highlight.max_analyzer_offset" request parameter with "plain" highlighter (opensearch-project#10919) (cherry picked from commit 5d327a2) Signed-off-by: Andrew Ross <[email protected]> * Change bwc test to properly reflect version Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Thomas Seidl <[email protected]>

) (opensearch-project#12376) (cherry picked from commit 43a2ce9) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ensearch-project#12373) (opensearch-project#12383) (cherry picked from commit 94f79a2) Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…#12374) (cherry picked from commit f5e1c4a) Signed-off-by: RS146BIJAY <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ect#11443) (opensearch-project#12154) * cleanup file cache on deleting index/shard directory * add index store listener --------- (cherry picked from commit c564ee3) Signed-off-by: panguixin <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pensearch-project#12391) (cherry picked from commit 10a6bc7) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…earch-project#12396) When this test fails, it takes [15 minutes to timeout][1]. There are many examples like this. While it generally succeeds after a retry or two, I think it is better to mute this test rather than observe regular failures that extend build times. [1]: https://build.ci.opensearch.org/job/gradle-check/33937/testReport/ (cherry picked from commit 6e702dc) Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pensearch-project#12393) Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 9b7541f)

…etails see: opensearch-project#12331 (opensearch-project#12375) (opensearch-project#12400) (cherry picked from commit 74589ef) Signed-off-by: Sorabh Hamirwasia <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…s flaky (opensearch-project#12415) (opensearch-project#12418) (cherry picked from commit bf050d4) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add bwc version 2.12.1 Signed-off-by: GitHub <[email protected]> * Fix version identifier for 2.12.1 Signed-off-by: Kunal Kotwani <[email protected]> --------- Signed-off-by: GitHub <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Kunal Kotwani <[email protected]>

…12077) (opensearch-project#12420) * Fix flaky test testScrollCreatedOnReplica * Disable scheduled refresh * Clean up segment collection assertions * Fix spotless --------- (cherry picked from commit 247e2ee) Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…project#12429) This test was muted due to an [intermediate state][1] during type removal. It now passes per my local testing. [1]: opensearch-project#2440 (comment) (cherry picked from commit a50d1f6) Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ject#12297) (opensearch-project#12425) * Add new method to pick a random replication strategy. * replace usage of refresh with refreshandWaitForReplication() * Add replication strategy logging. --------- (cherry picked from commit f8225ca) Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ensearch-project#11499) (opensearch-project#12151) * Allow composite aggregation under filter parent Composite aggregations are able to run under a filter aggregation or a reverse_nested aggreation, with no change required (other than not throwing an exception). Also cleaned up FilterAggregatorFactory a little. --------- (cherry picked from commit 24b0cc9) Signed-off-by: Michael Froh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ct#11626) (opensearch-project#12427) * Add support for scored named queries (opensearch-project#11626) Opensearch already support labelling the queries, that returns as a list in the returned results, of which query it matched. However one of the use case while doing hybrid search with query text and dense vector is to determine individual scores for each query type. This is very useful in further analysis and building offline model to generate better weights for ranking score. Hence adding this feature that sends the client to add the score for each matched query. --------- Signed-off-by: Dharin Shah <[email protected]> Signed-off-by: Dharin Shah <[email protected]> Co-authored-by: Dharin Shah <[email protected]> (cherry picked from commit 52b27f4) Signed-off-by: Andrew Ross <[email protected]> * Update version checks Signed-off-by: Andrew Ross <[email protected]> * Update test version guard Co-authored-by: Andriy Redko <[email protected]> Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: Dharin Shah <[email protected]> Signed-off-by: Dharin Shah <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Dharin Shah <[email protected]> Co-authored-by: Dharin Shah <[email protected]> Co-authored-by: Andriy Redko <[email protected]>

…ildSrc/src/testKit/thirdPartyAudit (opensearch-project#12464) (opensearch-project#12466)

…rch-project#12467) Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 24d7098)

…ribution/packages (opensearch-project#12461) (opensearch-project#12470) * Bump com.netflix.nebula.ospackage-base in /distribution/packages Bumps com.netflix.nebula.ospackage-base from 11.8.0 to 11.8.1. --- updated-dependencies: - dependency-name: com.netflix.nebula.ospackage-base dependency-type: direct:production update-type: version-update:semver-patch ... * Update changelog --------- (cherry picked from commit db4245d) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…ject#12462) (opensearch-project#12471) * Bump peter-evans/create-or-update-comment from 3 to 4 Bumps [peter-evans/create-or-update-comment](https://github.com/peter-evans/create-or-update-comment) from 3 to 4. - [Release notes](https://github.com/peter-evans/create-or-update-comment/releases) - [Commits](peter-evans/create-or-update-comment@v3...v4) --- updated-dependencies: - dependency-name: peter-evans/create-or-update-comment dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 0bd3ccd) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…search-project#11531) (opensearch-project#12474) * Fix get task API does not refresh resource stats (opensearch-project#11531) * Fix get task API does not refresh resource stats Signed-off-by: Gao Binlong <[email protected]> * modify change log Signed-off-by: Gao Binlong <[email protected]> --------- Signed-off-by: Gao Binlong <[email protected]> (cherry picked from commit 8d84eea) * Remove unused changelog Signed-off-by: Gao Binlong <[email protected]> --------- Signed-off-by: Gao Binlong <[email protected]>

…rch-project#12436) * Shard id awareness of SearchLookup Signed-off-by: Alexander Koval <[email protected]> * Add unit test for deprecated constructor Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: Alexander Koval <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Andrew Ross <[email protected]> (cherry picked from commit 87ac374) Co-authored-by: Alexander <[email protected]>

…roject#9470) (opensearch-project#12485) The `partSize()` value can be null if the underlying repository implementation does not implement file chunking. Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit c42ada8)

…pository chunk_size (opensearch-project#12277) (opensearch-project#12486) * implement logic of fetching blocks from multiple chunks of snapshot file. * Refactor and address comments. * apply spotless check * Address comments of using a different data structure to fetch blob parts. * remove unnecessary code. * Refactor outputstream usage. * refactor blobpart logic into a separate method and add unit tests. * Add new unit tests. --------- (cherry picked from commit 3125b94) Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ch-project#12307) (opensearch-project#12354) * Enable test suites to run with segrep * Refactor static settings into ParameterizedStaticSettingsOpenSearchIntegTestCase --------- (cherry picked from commit 181b985) Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

) (opensearch-project#12516) (cherry picked from commit 9814eb9) Signed-off-by: Jay Deng <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…t#12521) (opensearch-project#12524) * Bump lycheeverse/lychee-action from 1.9.1 to 1.9.3 Bumps [lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action) from 1.9.1 to 1.9.3. - [Release notes](https://github.com/lycheeverse/lychee-action/releases) - [Commits](lycheeverse/lychee-action@v1.9.1...v1.9.3) --- updated-dependencies: - dependency-name: lycheeverse/lychee-action dependency-type: direct:production update-type: version-update:semver-patch ... * Update changelog --------- (cherry picked from commit e498280) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…y-azure (opensearch-project#12520) (opensearch-project#12530) * Bump com.azure:azure-core in /plugins/repository-azure Bumps [com.azure:azure-core](https://github.com/Azure/azure-sdk-for-java) from 1.39.0 to 1.47.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-core_1.39.0...azure-core_1.47.0) --- updated-dependencies: - dependency-name: com.azure:azure-core dependency-type: direct:production update-type: version-update:semver-minor ... * Updating SHAs * Update changelog --------- (cherry picked from commit bd8af6a) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…g Ehcache as one of the pluggable disk cache option (opensearch-project#12514) Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]>

…opensearch-project#12287) (opensearch-project#12538) * Add kuromoji_completion analyzer and filter (opensearch-project#4835) * Use INDEX mode if an invalid value is set for mode in the kuromoji_completion filter --------- (cherry picked from commit 11836d0) Signed-off-by: Tatsuya Kawakami <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…es/hdfs-fixture (opensearch-project#12519) (opensearch-project#12534) * Bump ch.qos.logback:logback-core in /test/fixtures/hdfs-fixture Bumps [ch.qos.logback:logback-core](https://github.com/qos-ch/logback) from 1.2.13 to 1.5.3. - [Commits](qos-ch/logback@v_1.2.13...v_1.5.3) --- updated-dependencies: - dependency-name: ch.qos.logback:logback-core dependency-type: direct:production update-type: version-update:semver-minor ... * Update changelog --------- (cherry picked from commit ca827dd) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…rch-project#12551) Also includes removal from CODEOWNERS (cherry picked from commit 3056b67) Signed-off-by: Kartik Ganesh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ch-project#11679) (opensearch-project#12545) * Fix for deserialization bug in weighted round robin metadata --------- Signed-off-by: Anshu Agarwal <[email protected]> Co-authored-by: Anshu Agarwal <[email protected]>

…ctor Service and Emit IO Usage Stats opensearch-project#11880 (opensearch-project#12558) * Integrate IO Usage Tracker to the Resource Usage Collector Service and Emit IO Usage Stats (opensearch-project#11880) * Updating OS Version to 2_13 Signed-off-by: Ajay Kumar Movva <[email protected]>

…ensearch-project#12570) Signed-off-by: Harsha Vamsi Kalluri <[email protected]> (cherry picked from commit 8a22f28)

…arch-project#12565) (opensearch-project#12573) (cherry picked from commit 2b17902) Signed-off-by: Ashish Singh <[email protected]>

…pensearch-project#12588) * Bump codecov/codecov-action from 3 to 4 Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 3 to 4. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v3...v4) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-major ... * Update changelog --------- (cherry picked from commit cd6ce70) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…ildSrc/src/testKit/thirdPartyAudit/sample_jars (opensearch-project#12587) (opensearch-project#12592) * Bump org.apache.logging.log4j:log4j-core Bumps org.apache.logging.log4j:log4j-core from 2.23.0 to 2.23.1. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-core dependency-type: direct:production update-type: version-update:semver-patch ... * Update changelog --------- (cherry picked from commit 2be3898) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…ures/hdfs-fixture (opensearch-project#12580) (opensearch-project#12597) * Bump org.apache.zookeeper:zookeeper in /test/fixtures/hdfs-fixture Bumps org.apache.zookeeper:zookeeper from 3.9.1 to 3.9.2. --- updated-dependencies: - dependency-name: org.apache.zookeeper:zookeeper dependency-type: direct:production update-type: version-update:semver-patch ... * Update changelog --------- (cherry picked from commit 0c3aad7) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

Signed-off-by: Kunal Kotwani <[email protected]>

…repository-azure (opensearch-project#12579) (opensearch-project#12598) * Bump org.codehaus.woodstox:stax2-api in /plugins/repository-azure Bumps [org.codehaus.woodstox:stax2-api](https://github.com/FasterXML/stax2-api) from 4.2.1 to 4.2.2. - [Commits](FasterXML/stax2-api@stax2-api-4.2.1...stax2-api-4.2.2) --- updated-dependencies: - dependency-name: org.codehaus.woodstox:stax2-api dependency-type: direct:production update-type: version-update:semver-patch ... * Updating SHAs * Update changelog --------- (cherry picked from commit 246557c) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…hard causes opensearch jvm crashed (opensearch-project#12612) * onShardResult and onShardFailure are executed on one shard causes opensearch jvm crashed Signed-off-by: kkewwei <[email protected]> * unit test Signed-off-by: kkewwei <[email protected]> * spotlessJavaCheck Signed-off-by: kkewwei <[email protected]> * rename variable names Signed-off-by: kkewwei <[email protected]> * add changelog Signed-off-by: kkewwei <[email protected]> --------- Signed-off-by: kkewwei <[email protected]>

… files with classifiers (opensearch-project#12586) (opensearch-project#12613) Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 07e79e3)

…rch-project#12614) * Bump up jackson to 2.16.2 * Add changelog entry --------- (cherry picked from commit 69fc7dd) Signed-off-by: Aman Khare <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: Aman Khare <[email protected]> Co-authored-by: Aman Khare <[email protected]>

…2543) (opensearch-project#12605) * Fix PluginInfo bwc for opensearch_version field * Update how empty range list is handled --------- (cherry picked from commit a702f6a) Signed-off-by: Abhilasha Seth <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pensearch-project#12621) (cherry picked from commit 96dc820) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ch-project#12635) This line can be safely removed because the two lines following this assert test exactly the same condition. (cherry picked from commit ef9314e) Signed-off-by: Lukáš Vlček <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ards (opensearch-project#11269) (opensearch-project#12609) * Force merge API supports performing only on primary shards (opensearch-project#11269) * Force merge API supports performing on primary shards only Signed-off-by: Gao Binlong <[email protected]> * Modify change log Signed-off-by: Gao Binlong <[email protected]> * Fix test failure Signed-off-by: Gao Binlong <[email protected]> * Fix typo Signed-off-by: Gao Binlong <[email protected]> * Modify skip version Signed-off-by: Gao Binlong <[email protected]> * Add version check and more tests Signed-off-by: Gao Binlong <[email protected]> * Format code Signed-off-by: Gao Binlong <[email protected]> * Modify supported version and add more test Signed-off-by: Gao Binlong <[email protected]> * Change the supported version to 3.0.0 Signed-off-by: Gao Binlong <[email protected]> * Add test case in SegmentReplicationIT Signed-off-by: Gao Binlong <[email protected]> * Optimize the test code Signed-off-by: Gao Binlong <[email protected]> --------- Signed-off-by: Gao Binlong <[email protected]> (cherry picked from commit e6eec36) * Modify supported version to 2.13.0 Signed-off-by: Gao Binlong <[email protected]> * Fix test failure Signed-off-by: Gao Binlong <[email protected]> * Fix yml test failure Signed-off-by: Gao Binlong <[email protected]> * Fix yaml test failure Signed-off-by: Gao Binlong <[email protected]> * Fix typo and test failure Signed-off-by: Gao Binlong <[email protected]> --------- Signed-off-by: Gao Binlong <[email protected]>

opensearch-project#12636) Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 949f316)

…query is functionally match-all for a segment (opensearch-project#12629) Quickly compute terms aggregations when the top-level query is functionally match-all for a segment (opensearch-project#11643) --------- Signed-off-by: Sandesh Kumar <[email protected]>

…roject#12646) * Tracing for deep search path Signed-off-by: David Zane <[email protected]> * Refactor search phase tracing instrumentation to detach span creation from the SearchRequestOperationsListener Signed-off-by: Andriy Redko <[email protected]> * Close WrappedPhase span Signed-off-by: David Zane <[email protected]> * Add asserting listener class Signed-off-by: David Zane <[email protected]> * Cleanup AbstractSearchAsyncActionTests tests Signed-off-by: Andriy Redko <[email protected]> * Cleanup SearchAsyncActionTests tests Signed-off-by: Andriy Redko <[email protected]> * Cleanup CanMatchPreFilterSearchPhaseTests tests Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: David Zane <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: Andriy Redko <[email protected]> (cherry picked from commit 77a1193)

…pensearch-project#12655) * Upgrade aws-sdk version to 2.20.86 * Update SHAs Signed-off-by: Sachin Kale <[email protected]>

…rch-project#12634) Signed-off-by: Ankit Kala <[email protected]>

…pensearch-project#12583) (opensearch-project#12659) Signed-off-by: Ajay Kumar Movva <[email protected]>

…amework (opensearch-project#12583) (opensearch-project#12659)" (opensearch-project#12671) This reverts commit 3970ec4. Reverting as this introduced test failures detailed in opensearch-project#12664. Signed-off-by: Andrew Ross <[email protected]>

…opensearch-project#12668) Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit ce43f30)

…s are reused for replication (opensearch-project#11786) (opensearch-project#11806) * Use SegmentReplicationTarget test to validate non-active on-disk files are reused for replication * Fix original shard level unit test --------- (cherry picked from commit fe98aad) Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ntrolled by a feature flag (opensearch-project#12533) (opensearch-project#12600) * Adding changelog * Fixing gradle build issue * Fixing CacheService test * Adding UT in IndicesRequestCache with feature flag for more coverage * Updating changelog and renaming feature flag setting * Moving feature flag setting handling logic to CacheService by maintaining backward compatibility * Fixing broken UTs --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]>

…ct#12337) (opensearch-project#12359) * Avoid string out of bounds error in snapshot delete Test failure opensearch-project#8771 shows cases where certain random seeds trigger this case. The bug is clear: the substring() call should happen after the startsWith() check in case the blob name is shorter than the prefix length being used as the start index of the substring call. I don't yet know if/how this manifests in real deployments. * Extract common UUID parsing method --------- (cherry picked from commit f86bfa8) Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…on during fetch… (opensearch-project#12654) * Removing unused fetch sub phase processor initialization during fetch… (opensearch-project#12503) * Removing unused fetch sub phase processor initialization during fetch phase execution Signed-off-by: Ankit Jain <[email protected]> * Updating CHANGELOG.md with the bug fix Signed-off-by: Ankit Jain <[email protected]> * Fixing smoke test failures Signed-off-by: Ankit Jain <[email protected]> * Addressing review comments Signed-off-by: Ankit Jain <[email protected]> * Addressing review comments around boolean check convention Signed-off-by: Ankit Jain <[email protected]> --------- Signed-off-by: Ankit Jain <[email protected]> (cherry picked from commit da5b205) * Removing unintended CHANGELOG issue Signed-off-by: Ankit Jain <[email protected]> --------- Signed-off-by: Ankit Jain <[email protected]>

…-project#12319) (opensearch-project#12677) (cherry picked from commit b265215) Signed-off-by: Harish Bhakuni <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…2650) (opensearch-project#12679) * Fixing flaky tiered cache test * Removing unnecessary comment * Removing unused variable --------- Signed-off-by: Sagar Upadhyaya <[email protected]>

…rch-project#12672) (opensearch-project#12682) * Simplify remote directory cleanup after snapshot delete to avoid concurrent cleanup task runs for same shard. * Address PR Comments. --------- (cherry picked from commit d0467b3) Signed-off-by: Harish Bhakuni <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Harish Bhakuni <[email protected]>

…pensearch-project#12577) (opensearch-project#12684) * handle unexpected exception on success callback of translog upload (cherry picked from commit 4a0feee) Signed-off-by: Varun Bansal <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…2642) (opensearch-project#12692) * Adds support for asynchronous gauge metric type * Adds change log * incorporate pr comments * fixes build errors --------- Signed-off-by: Gagan Juneja <[email protected]> Signed-off-by: Gagan Juneja <[email protected]> Co-authored-by: Gagan Juneja <[email protected]>

…bled Integration Tests (opensearch-project#12487) (opensearch-project#12647) * Add support for parameterized remote-store testing for Integration Tests. * Rename variable names. --------- (cherry picked from commit 8ce3687) Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Rishikesh Pasham <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ensearch-project#12562) (opensearch-project#12623) * Mute flaky test testMultiGetWithNetworkDisruption_FailOpenEnabled * mute flaky test testShardRoutingWithNetworkDisruption_FailOpenEnabled * spotless fix --------- (cherry picked from commit 1e9c05c) Signed-off-by: Anshu Agarwal <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Anshu Agarwal <[email protected]>

…ld (opensearch-project#12676) (opensearch-project#12681) Fixes opensearch-project#10253 (cherry picked from commit fcecd00) Signed-off-by: Michael Froh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ator (opensearch-project#12653) (opensearch-project#12698) * [Tiered caching] Supporting removal function on EhcacheDiskCache iterator * Minor refactoring in unit test --------- (cherry picked from commit ec61cbc) Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ct#12542) (opensearch-project#12696) * [Tiered Caching] Cache tier policies (opensearch-project#12542) * Adds policy interface and took time policy impl Signed-off-by: Peter Alfonsi <[email protected]> * Changes IndicesService to write a CachePolicyInfoWrapper before the QSR Signed-off-by: Peter Alfonsi <[email protected]> * Moved took time logic from QSR to IndicesService Signed-off-by: Peter Alfonsi <[email protected]> * spotlessApply Signed-off-by: Peter Alfonsi <[email protected]> * Addressed ansjcy's comments Signed-off-by: Peter Alfonsi <[email protected]> * Partial rebase on most recent changes Signed-off-by: Peter Alfonsi <[email protected]> * Integrated policies with new TSC changes Signed-off-by: Peter Alfonsi <[email protected]> * Reverted unintended change to idea/vcs.xml Signed-off-by: Peter Alfonsi <[email protected]> * javadocs Signed-off-by: Peter Alfonsi <[email protected]> * github actions Signed-off-by: Peter Alfonsi <[email protected]> * Set default threshold value to 10 ms Signed-off-by: Peter Alfonsi <[email protected]> * Addressed Sorabh's comments Signed-off-by: Peter Alfonsi <[email protected]> * Addressed Sorabh's second round of comments Signed-off-by: Peter Alfonsi <[email protected]> * Set cachedQueryParser in IRC Signed-off-by: Peter Alfonsi <[email protected]> * Addressed Sorabh's comments besides dynamic setting Signed-off-by: Peter Alfonsi <[email protected]> * Removed dynamic setting, misc comments Signed-off-by: Peter Alfonsi <[email protected]> * Added changelog entry Signed-off-by: Peter Alfonsi <[email protected]> * Added missing javadoc Signed-off-by: Peter Alfonsi <[email protected]> * Fixed failed gradle run Signed-off-by: Peter Alfonsi <[email protected]> * Added setting validation test Signed-off-by: Peter Alfonsi <[email protected]> * rerun gradle for flaky IT Signed-off-by: Peter Alfonsi <[email protected]> * javadocs Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> (cherry picked from commit ccdf3ff) * rerunning gradle Signed-off-by: Peter Alfonsi <[email protected]> * rerunning gradle Signed-off-by: Peter Alfonsi <[email protected]> * rerunning gradle Signed-off-by: Peter Alfonsi <[email protected]> * rerunning gradle Signed-off-by: Peter Alfonsi <[email protected]> * rerunning gradle Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]>

…p for shallow snapshot deletion. (opensearch-project#12701) (opensearch-project#12703) (cherry picked from commit 48004ee) Signed-off-by: Harish Bhakuni <[email protected]>

…pensearch-project#12702) (opensearch-project#12716) Signed-off-by: Ajay Kumar Movva <[email protected]>

…ent (opensearch-project#12625) (opensearch-project#12707) * [Tiered Caching] Indices Request cache stalekey management (opensearch-project#12625) * Introduce IndicesRequestCacheCleanupManager Signed-off-by: Kiran Prakash <[email protected]> * using cleanup mgr to enqueue cleanups Signed-off-by: Kiran Prakash <[email protected]> * readability improvements Signed-off-by: Kiran Prakash <[email protected]> * update updateCleanupKeyToCountMap on new cache entry Signed-off-by: Kiran Prakash <[email protected]> * create IndicesRequestCacheCleanupManager & settings and validators Signed-off-by: Kiran Prakash <[email protected]> * Add IRC specific cache cleaner and remove from IndicesService Signed-off-by: Kiran Prakash <[email protected]> * Unit Tests Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * move cachecleaner inside mgr Signed-off-by: Kiran Prakash <[email protected]> * remove processCleanupKeys Signed-off-by: Kiran Prakash <[email protected]> * minor cleanups Signed-off-by: Kiran Prakash <[email protected]> * add updateCleanupKeyToCountMapOnCacheEviction Signed-off-by: Kiran Prakash <[email protected]> * remove locks and make all methods synchronized Signed-off-by: Kiran Prakash <[email protected]> * spotless Signed-off-by: Kiran Prakash <[email protected]> * updateCleanupKeyToCountMapOnCacheEviction Signed-off-by: Kiran Prakash <[email protected]> * Testing Signed-off-by: Kiran Prakash <[email protected]> * add Reschedule back to indices service Signed-off-by: Kiran Prakash <[email protected]> * rename updateStaleKeysCount to incrementStaleKeysCount Signed-off-by: Kiran Prakash <[email protected]> * rename getStaleKeysCountForTesting to getStaleKeysCount Signed-off-by: Kiran Prakash <[email protected]> * rename threshold to stalenessThreshold Signed-off-by: Kiran Prakash <[email protected]> * check for cleanupKey.entity == null Signed-off-by: Kiran Prakash <[email protected]> * use computeIfPresent with keycountmap Signed-off-by: Kiran Prakash <[email protected]> * log both staleKeysInCache & Staleness in debug logs Signed-off-by: Kiran Prakash <[email protected]> * Use HashMap instead of ConcurrentMap Signed-off-by: Kiran Prakash <[email protected]> * Address b/w compatibility Signed-off-by: Kiran Prakash <[email protected]> * remove synchronized for updateCleanupKeyToCountMapOnCacheEviction Signed-off-by: Kiran Prakash <[email protected]> * make cleanCache synchronized Signed-off-by: Kiran Prakash <[email protected]> * remove shouldRemoveKey Signed-off-by: Kiran Prakash <[email protected]> * spotlessApply Signed-off-by: Kiran Prakash <[email protected]> --------- Signed-off-by: Kiran Prakash <[email protected]> (cherry picked from commit b4da802) * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <[email protected]> --------- Signed-off-by: Kiran Prakash <[email protected]>

…ation (opensearch-project#12700) (opensearch-project#12749) --------- (cherry picked from commit 902a10b) Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ion (opensearch-project#11986) (opensearch-project#12751) * Introducing mixed mode support for remote store migration --------- Signed-off-by: Gaurav Bafna <[email protected]>

…earch-project#12712) * Decouple remote state configuration (cherry picked from commit 00d4b71) Signed-off-by: Sooraj Sinha <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…bledSanityIT.testGauge (opensearch-project#12745) (opensearch-project#12747) (cherry picked from commit 2567792) Signed-off-by: Gagan Juneja <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Gagan Juneja <[email protected]>

…nsearch-project#12756) (cherry picked from commit 7a6f8b0) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

opensearch-project#12505) (opensearch-project#12754) (cherry picked from commit 18e5816) Signed-off-by: Lakshya Taragi <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ch-project#12757) * Async Fetcher class changes (cherry picked from commit 63f4f13) Signed-off-by: Gaurav Chandani <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…roject#12709) (opensearch-project#12736) * [Tiered Caching] Serializers for ehcache (opensearch-project#12709) Adds serializers and integrates them into ehcache disk cache --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]> (cherry picked from commit 21b28f2) * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Removed incorrect changelog line Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> * Rerunning gradle checks Signed-off-by: Peter Alfonsi <[email protected]> --------- Signed-off-by: Peter Alfonsi <[email protected]> Co-authored-by: Peter Alfonsi <[email protected]>

…ect#12631) (opensearch-project#12750) Signed-off-by: mgodwan <[email protected]>

… calls (opensearch-project#9760) (opensearch-project#12762) * PrimaryShardAllocator refactor to abstract out shard state and method calls (cherry picked from commit afd3969) Signed-off-by: Shivansh Arora <[email protected]> Signed-off-by: Shivansh Arora <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…own (opensearch-project#12734) (opensearch-project#12760) * Adding logic to clear up the disk cache files during close() * Adding logic to update entries count after invalidateAll() * Removing unneeded system log statement * Added comment in test for readability * Fixing issue where we were sending compacted byte[] array to ehcache but calculating size with padded byte[] --------- Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: Sagar <[email protected]>

…ng strategies (opensearch-project#12441) (opensearch-project#12761) * Abstract AsyncShardFetch cache to allow restructuring for batch mode Signed-off-by: Aman Khare <[email protected]> (cherry picked from commit b15cb0c)

…primary shards (opensearch-project#12759) * Added transport action for bulk async shard fetch for primary shards (opensearch-project#8218) * Added transport action for bulk async shard fetch for primary shards Signed-off-by: Shivansh Arora <[email protected]> (cherry picked from commit 8cb4bb4) Signed-off-by: Shivansh Arora <[email protected]>

opensearch-project#11699) (opensearch-project#12765) (cherry picked from commit d4e1ab1) Signed-off-by: Sooraj Sinha <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…roject#8218 (opensearch-project#8356) (opensearch-project#12770) * Add batch async shard fetch transport action for replica Signed-off-by: Shivansh Arora <[email protected]> (cherry picked from commit 12115d1)

… (opensearch-project#12789) (opensearch-project#12794) * [FEATURE] Built-in secure transports support Signed-off-by: Andriy Redko <[email protected]> * Added more tests, addressing code review comments Signed-off-by: Andriy Redko <[email protected]> * Address code review comments Signed-off-by: Andriy Redko <[email protected]> * Address code review comments Signed-off-by: Andriy Redko <[email protected]> * Address code review comments Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: Andriy Redko <[email protected]> (cherry picked from commit 773a939) (cherry picked from commit 3cc154a)

…pensearch-project#8776) (opensearch-project#12773) (opensearch-project#12810) * BaseGatewayShardAllocator changes for Assigning the batch of shards (cherry picked from commit ef50fb4) (cherry picked from commit 41d11e1) Signed-off-by: Gaurav Chandani <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Aman Khare <[email protected]>

…ch of shards (opensearch-project#8916) (opensearch-project#12813) (opensearch-project#12823) * Add PrimaryShardBatchAllocator to take allocation decisions for a batch of shards (cherry picked from commit a499d1e) (cherry picked from commit b2d22d4) Signed-off-by: Aman Khare <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…uster-state from remote (opensearch-project#12252) (opensearch-project#12825) Signed-off-by: Rajiv Kumar Vaidyanathan <[email protected]>

…s. (opensearch-project#12496) (opensearch-project#12832) * Integrate with CPU admission controller for cluster-manager Read API's. The admission control is enforced at the transport layer. Signed-off-by: Rajiv Kumar Vaidyanathan <[email protected]>

…roject#12846) (cherry picked from commit 0c0bcd9) Signed-off-by: Varun Bansal <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…#12851) (opensearch-project#12856) (cherry picked from commit 122e944) Signed-off-by: Varun Bansal <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ene writer (opensearch-project#12710) (opensearch-project#12786) (opensearch-project#12857) (cherry picked from commit c369ec4) (cherry picked from commit 29bcc48) Signed-off-by: Prabhat Sharma <[email protected]> Co-authored-by: Prabhat Sharma <[email protected]>

…t#12874) GetTermVersion action has been added to check if cluster state needs to be updated as a performance improvement on all cluster state checks. The Authorization systems in the Security Plugin check all actions that do not start with `internal:` for permissions causing users without `cluster:monitor/*` permissions to start getting 403 exceptions. This change signals that this action does not require an authorization check by any security systems since it cannot be called via REST APIs. Signed-off-by: Peter Nied <[email protected]>

…pensearch-project#12890) (cherry picked from commit 90b2e4d) Signed-off-by: Gao Binlong <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

The 2.13 release notes have been created. Signed-off-by: Andrew Ross <[email protected]>

* Update CODEOWNERS * Update README.md and SECURITY.md * Add Wazuh configuration files * Update README.md Signed-off-by: Álex Ruiz <[email protected]>

Signed-off-by: Álex Ruiz <[email protected]>

* Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Update template settings --------- Signed-off-by: Álex Ruiz <[email protected]>

* Add ECS mappings generator, documentation and files for vulnerability detector * Add event generator script * Add default query fields --------- Signed-off-by: Álex Ruiz <[email protected]>

Signed-off-by: Álex Ruiz <[email protected]>

* Fix bug with -i option * Improve error handling

* Improve ISM init script * Change log file path

* Update config files * Add VERSION file

* Ignore artifacts folder * Update build script - Updated to v2.11.0 version. - Skipped compilation of the plugins - The artifact nameis sent to a text file, to access it easily in GitHub Actions. * Add GH action to build min packages * Remove commented code * Remove unused code

* Add very basic Docker environment That will do for now * Add latest changes * Update Docker environment - Remove build.md which was included by mistake. - Improve dev.sh script. - Update .gitignore to exclude artifacts folder. - Create .dockerignore file. - Replace get_version.sh script with inline command. - Reduce image size by using alpine as base image. --------- Signed-off-by: Álex Ruiz <[email protected]>

* Rename packages to wazuh-indexer * Include VERSION file into packages * Apply Wazuh version to packages names * Improve build.sh script Apply suggestions from ShellCheck

* Remove 'events' ECS field * Add 'wazuh' custom field * Update event_generator.py for vulnerability detector

Updates the script to upload the wazuh-template.json to the indexer. Signed-off-by: Álex Ruiz <[email protected]>

* Add script to assemble arm64 and x64 archives (tar) * Cleanup * Update config file with latest upstream changes * Change packages maintainer information * Fix wrong substitution of config files * Update dockerignore to ignore git folder * Update wazuh-indexer.rpm.spec Remove unnecessary echo commands * Add wazuh-indexer-performance-analyzer.service Required to assembly RPM. The plugin does not install this file, so it needs to be added manually. * Update assemble.sh Successfully assemble RPM x64. Runner needed to arm64 * Update `build.yml` * Add WIP documentation for packages' generation * Test new approach using reusable workflows * Fix errors * Restructure reusable workflow * Fix upload and download paths * New try - Adds a reusable workflow to return the version of Wazuh set in source code. - Attempt to dynamically generate artifacts name to normalize them for usage between jobs. - Adds revision as input for the workflow. - Cleanup * Emulate assemble to test upload of the reusable assembly workflow * Add Caching Gradle dependencies * Remove extra '-' in the packages names on the assembly job * Final cleanup * Enable RPM package assemble Remove unused code * Fix regex to get package name * Fix download-artifact destination path * Exclude unimplemented deb assembly Extend example to run with Act

* Add template and settings to disable replicas on ISM plugin internal indices * Fix documentation Replaces exit 1 statements with return 1 * Fix uncommented comment line

* Update ism-init script to parametrize the path of the wazuh-template --------- Signed-off-by: Álex Ruiz <[email protected]>

* Add tools to assemble DEB packages * Move wazuh-indexer-performance-analyzer.service to common * Enable assembly of DEB packages * Enable full set of plugins * Actually skip tar assembly * Add installation of dependencies for DEB assembly * Install dependencies using sudo * Format files * Refactor assemble script

Signed-off-by: Álex Ruiz <[email protected]>

* Build scripts and GH workflows artifacts naming fix * Add git to dev docker image * Fixing jobs' inputs and outputs * remove name input from r_assemble.yml * Setting qualifier to 1 when not specified * Add revision flag to scripts and workflow * Fix copying of packages at assemble.sh * Use suffix variable instead of architecture * Fix suffix name in assemble.sh * Mix solutions to comply with the package naming convention * Remove unused code * Use correct name for assembled package Remove code no longer needed * Remove outdated comments --------- Co-authored-by: Álex Ruiz <[email protected]>

* Switching to short SHA commit form in package names Signed-off-by: Fede Tux <[email protected]> * Update r_commit_sha.yml Signed-off-by: Federico Gustavo Galland <[email protected]> * Update r_commit_sha.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Fede Tux <[email protected]> Signed-off-by: Federico Gustavo Galland <[email protected]> Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Álex Ruiz <[email protected]>

* add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]>

* add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]>

* Switch /var/run references to /run * Remove unneeded files from assembled packages (#115) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment --------- Co-authored-by: Álex Ruiz <[email protected]> * Add missing tools and files back into Wazuh Indexer packages (#117) * add remove files function to assemble.sh * Remove unneeded files on assembled tar packages * Remove duplicated function Fix wrong variable assignment * Adding function to package Wazuh`s tools to assemble.sh * Make the files' versions follow the repo's VERSION file * Fix download of Wazuh tools for packages assembly --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Remove unneeded symbolic links from assembled packages (#121) * Remove reference to install_demo_configuration.sh --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]>

Runs the workflow on pull request changes

* Add function to look for VERSION in the correct path * Update assemble.sh Adds wget as dependency * Download files using curl instead of wget * Update assemble.sh Revert assembly with minimal plugins for testing Signed-off-by: Álex Ruiz <[email protected]> * Add Dockerfile and docker-compose for the package assembly stage * Assemble packages with minimal plugin set when "test" variable is set to "true" * Update README with assemble.sh docker image * Fixing env variable naming convention and removing wget dependency * Improve Docker environments Adds environments to build packages * Fix small typos * More fixes * Add documentation * Adding -p flag to mkdir so it doesnt fail when the folder is already present * Format files --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]>

…#130) Co-authored-by: Álex Ruiz <[email protected]>

* Download wazuh-template.json from wazuh/wazuh repo * Add wazuh-template.json to RPM package spec * Setting wazuh-template.json attributes to 660 * Change wazuh-template.json attributes in debmake_install.sh * Put template download command within a function * Small fixes and format * Apply correct file permissions to the wazuh-template.json --------- Co-authored-by: Álex Ruiz <[email protected]>

* Adding debian packaging config files from Opensearch * Copy debian/ folder to the build dir for debmake to parse * Remove redundant steps from debian/postinst --------- Co-authored-by: Álex Ruiz <[email protected]>

* Run workflow on push * Set build workflow inputs to required * Normalize the use of quotes for the build workflow inputs * Add ternary operator * Add missing ternary operator

* Fine tuning permissions on RPM spec file * Get plugins using maven * Rolling back changes to spec file * Format files --------- Co-authored-by: Álex Ruiz <[email protected]>

* Add new custom field to the vulnerability detector index * Update event generator tool * Remove base.labels ECS field from wazuh-states-vulnerabilities index mappings

* Fine tuning permissions on RPM spec file * Build a list of files to be packaged excluding items that need special permissions * Fix bad permissions on directories * Remove system directories from packaging definition * Changing permissions on deb packages * Skip unneeded dh_fixperms stage in debian/rules * Clean & format --------- Co-authored-by: Álex Ruiz <[email protected]>

* Init. Amazon Security Lake integration Signed-off-by: Álex Ruiz <[email protected]>

* Add events generator tool for wazuh-alerts * Fix typo in README.md Signed-off-by: Álex Ruiz <[email protected]> * Make timestamps timezone aware --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]>

Signed-off-by: Álex Ruiz <[email protected]>

* Adding Python script that receives a continuous json stream over stdin and outputs parquet to Security Lake * Adding logstash pipeline for python script * encode_parquet() function fixed to handle lists of dictionaries * Correct error in encode_parquet() * Avoid storing the block ending in the output buffer * Add comments on handling files and streams with pyarrow for future reference * Add s3 handling reference links * Write parquet directly to bucket * Added basics of map_to_ocsf() function * Minor fixes * Map alerts to OCSF as they are read * Add script to convert Wazuh events to OCSF Also adds a simple test script * Add OCSF converter + Parquet encoder + test scripts * Update .gitignore * Include the contents of the alert under unmapped * Add support for different OCSF schema versions * Use custom ocsf module to map alerts * Modify script to use converter class * Code polish and fix errors * Remove unnecessary type declaration from debug flag * Improved parquet encoding * Initial commit for test env's docker-compose.yml * Remove sudo references from docker-compose.yml * Add operational Python module to transform events to OCSF * Create minimal Docker environment to test and develop the integration. * Fix events-generator's Inventory starvation * Remove files present in #147 * Cleanup * Add FQDN hostnames to services for certificates creation * Add S3 Ninja (Mock) (#165) * Setup certificates in Wazuh Indexer and Logstash containers (#166) * Add certificate generator service * Add certificate config to docker compose file * Use secrets for certificates * Disable permission handling inside cert's generator entrypoint.sh * Back to using a bind mount for certs * Have entrypoint.sh generate certs with 1000:1000 ownership * Correct certificate permissions and bind mounting * Add security initialization variable to compose file * Fix permissions on certs generator entrypoint * Add cert generator config file * Remove old cert generator dir * Set indexer hostname right in pipeline file * Roll back commented code --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Álex Ruiz <[email protected]> * Fix Logstash pipelines * Remove unused file * Implement OCSF severity normalize function --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]> Co-authored-by: Federico Gustavo Galland <[email protected]>

* Attemtp to automate package's testing * Fix typo * Update setup gradle action * Remove file from another PR * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> --------- Signed-off-by: Álex Ruiz <[email protected]>

* Update vulnerability-states fields Adds wazuh.schema.version * Update events generator

* Attemtp to automate package's testing * Fix typo * Add sudo * Split test steps and manage errors * Add --no-pager to journalctl * Add certs generator * Improve error handling * Update r_test.yml Fix indentation Signed-off-by: Álex Ruiz <[email protected]> * Fix error handling * Add testing of RPM packages * Improve multi-os testing * Add TEST env var * Add braces to if conditionals * Remove all curly braches from if conditionals * braces again * Install RPM package in Docker * Remove sudo for RPM installation * Bind artifacts/dist to RPM docker test container * Bind artifacts/dist to RPM docker test container * Avoid prompt during yum install * Fix bind volume --------- Signed-off-by: Álex Ruiz <[email protected]>

* Attemtp to automate package's testing * Add workflow file to upload packages to S3 * Skip testing to test whether the upload works * Fix package names * Fix upload workflow name * Pass secrets to the reusable workflow * Fix indentation * Fix indentation * Remove test workflow from this PR * Add boolean input to control when the package is uploaded to the S3 bucket * [UI/UX] Improve inputs description --------- Signed-off-by: Álex Ruiz <[email protected]>

* Update wazuh-states-vulnerabilities index mapping * Extend ECS Vulnerability fields

* Add script to get the version of OpenSearch * Set revision to 0 by default. - Reduce inputs for scripts. - Add script to generate packages' naming convention. - Make scripts self-aware of the OpenSearch version. * Fix assemble * Smoke test new pipeline to build packages * Fix syntax errors * Update build.yml Signed-off-by: Álex Ruiz <[email protected]> * Add workflow to build packages on push * Run actionlint * Fix jq argjson * Fix set matrix output ? * Try new approach using a single workflow * Fix GITHUB_OUTPUT * Fix baptizer invocation * Add testing and upload to new approach * Fix hard coded revision number on RPM assembly * New attempt * Skip upload unless specified * Install plugins on RPM * Promote new approach Removes previous workflows to generate packages * Fix workflow name * Attempt to fix release package naming * Fix build.sh invocation from workflow * Use min package name in workflow * Use min package name for release naming convention in workflow * Attemtp to fix regex * Upgrade to aws-actions/configure-aws-credentials@v4 Clean up * Apply latest requirements Add workflow with single matrix for QA use. Rename inputs. Add checksum input. * Add checksum generation and upload * Use choice as input types for system and architecture * Invoke build single packages with upload option * Add documentation and clean up * Rename scripts folder to packaging_scripts --------- Signed-off-by: Álex Ruiz <[email protected]>

* Assemble tar packages * Add files to generate Docker images First working version * Fix certs path * clean up * Working indexer in Docker * Add documentation to build Docker images Simplify names of Docker build args * Remove unused Docker dependencies --------- Signed-off-by: Álex Ruiz <[email protected]>

Allows invocation usin the GH API

@timestamp

* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line

@timestamp

* Migrate from #147 * Update amazon-security-lake integration - Improved documentation. - Python code has been moved to `wazuh-indexer/integrations/amazon-security-lake/src`. - Development environment now uses OpenSearch 2.12.0. - The `wazuh.integration.security.lake` container now displays logs, by watching logstash's log file. - [**NEEDS FIX**] As a temporary solution, the `INDEXER_USERNAME` and `INDEXER_PASSWORD` values have been added as an environment variable to the `wazuh.integration.security.lake` container. These values should be set at Dockerfile level, but isn't working, probably due to permission denied on invocation of the `setup.sh` script. - [**NEEDS FIX**] As a temporary solution, the output file of the `indexer-to-file` pipeline as been moved to `/var/log/logstash/indexer-to-file`. Previous path `/usr/share/logstash/pipeline/indexer-to-file.json` results in permission denied. - [**NEEDS FIX**] As a temporary solution, the input.opensearch.query has been replaced with `match_all`, as the previous one does not return any data, probably to the use of time filters `gt: now-1m`. - Standard output enable for `/usr/share/logstash/pipeline/indexer-to-file.json`. - [**NEEDS FIX**] ECS compatibility disabled: `echo "pipeline.ecs_compatibility: disabled" >> /etc/logstash/logstash.yml` -- to be included automatically - Python3 environment path added to the `indexer-to-integrator` pipeline. * Disable ECS compatibility (auto) - Adds pipeline.ecs_compatibility: disabled at Dockerfile level. - Removes `INDEXER_USERNAME` and `INDEXER_PASSWORD` as environment variables on the `wazuh.integration.security.lake` container. * Add @timestamp field to sample alerts * Fix Logstash pipelines * Add working indexer-to-s3 pipeline * Add working Python script up to S3 upload * Add latest changes * Remove duplicated line * Add working environment with minimal AWS lambda function * Mount src folder to Lambda's workdir * Add first functional lambda function Tested on local environment, using S3 Ninja and a Lambda container * Working state * Add documentation * Improve code * Improve code * Clean up * Add instructions to build a deployment package * Make zip file lighter * Use default name for aws_region * Add destination bucket validation * Add env var validation and full destination S3 path * Add AWS_ENDPOINT environment variable * Rename AWS_DEFAULT_REGION * Remove unused env vars * Remove unused file and improve documentation a bit. * Makefile improvements * Use dummy env variables --------- Signed-off-by: Álex Ruiz <[email protected]>

* Map events to OCSF's Security Finding class * Improve models (inheritance). Add OCSF_CLASS env variable * Move constants to the models * Fix validation error

* Added id input * Changed name to run-name

…kages accordingly (#231)

* Add documentation for the Amazon Security Lake integration * Add images via upload Signed-off-by: Álex Ruiz <[email protected]> * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Use jpeg * Add files via upload Signed-off-by: Álex Ruiz <[email protected]> * Fix some typos * Add CONTRIBUTING.md * Apply improvements to the ASL docu --------- Signed-off-by: Álex Ruiz <[email protected]>

* Add integration with Elastic Draft * Update Elastic integration Draft * Add Elastic integration folder Draft * Changing the kibana system user * Add Elastic integration Working --------- Co-authored-by: Fede Tux <[email protected]>

* Added S3 URI output * Added ID input and S3 URI output * Improved workflow run name * Added name statement * Added name statement * Removed file * Added ID input description * Update build.yml --------- Co-authored-by: Álex Ruiz <[email protected]>

* Add docker environment * Add README Move files to the corresponding folde * Enable TLS in dashboards --------- Co-authored-by: Álex Ruiz <[email protected]>

* Add Splunk integration Draft * Fix certificate errors * Add cfssl container to generate and sign splunk certs * Add cfssl configuration fiels * Update Splunk integration --------- Signed-off-by: Álex Ruiz <[email protected]> Co-authored-by: Fede Tux <[email protected]>

* Init commit [DRAFT] Adds a Compose environment * Mount alerts as shared volume instead of file * Update documentation and clean up files --------- Co-authored-by: Fede Tux <[email protected]>

* Add Manager to OpenSearch integreation Also fixes small issues on other integrations * Add changes to README

* Testy test test * Update artifact name Skip lintian * Update Mantainers for Debian package metadata

* Remove references to indexer-ism-init.sh and wazuh-template.json * Roll back remaining content from ISM rollover+alias feature * Remove commented code --------- Co-authored-by: Álex Ruiz <[email protected]>

Ignoring /run and /var/run when building directories into RPM packages. #285

Ignoring /run and /var/run when building directories into RPM packages. #285

Commits on Jan 30, 2024

Commits on Jan 31, 2024

Commits on Feb 2, 2024

Commits on Feb 3, 2024

Commits on Feb 5, 2024

Commits on Feb 6, 2024

Commits on Feb 7, 2024

Commits on Feb 8, 2024

Commits on Feb 9, 2024

Commits on Feb 10, 2024

Commits on Feb 12, 2024

Commits on Feb 14, 2024

Commits on Feb 15, 2024

Commits on Feb 16, 2024

Commits on Feb 17, 2024

Commits on Feb 19, 2024

Commits on Feb 20, 2024

Commits on Feb 21, 2024

Commits on Feb 22, 2024

Commits on Feb 23, 2024

Commits on Feb 26, 2024

Commits on Feb 27, 2024

Commits on Feb 28, 2024

Commits on Feb 29, 2024

Commits on Mar 3, 2024

Commits on Mar 4, 2024

Commits on Mar 5, 2024

Commits on Mar 7, 2024

Commits on Mar 8, 2024

Commits on Mar 9, 2024

Commits on Mar 11, 2024

Commits on Mar 12, 2024

Commits on Mar 13, 2024

Commits on Mar 14, 2024

Commits on Mar 15, 2024

Commits on Mar 16, 2024

Commits on Mar 18, 2024

Commits on Mar 19, 2024

Commits on Mar 20, 2024

Commits on Mar 21, 2024

Commits on Mar 22, 2024

Commits on Mar 25, 2024

Commits on Apr 24, 2024

Commits on Apr 26, 2024

Commits on Apr 29, 2024

Commits on Apr 30, 2024

Commits on May 17, 2024

Commits on May 21, 2024

Commits on May 28, 2024

Commits on May 29, 2024

Commits on May 31, 2024

Commits on Jun 5, 2024

Commits on Jun 7, 2024

Commits on Jun 11, 2024

Commits on Jun 12, 2024

Commits on Jun 14, 2024

Commits on Jun 25, 2024