[Release 3.0] Planned Breaking Changes for 3.0 in Plugin

[peterzhuamazon]

[Release 3.0] Planned Breaking Changes for 3.0 in Plugin

As mentioned in this META Issue, we want to track and increase transparency around the breaking changes planned for the 3.0 release across participating plugins.

Please kindly document all planned breaking changes related to your plugin for 3.0 here, by sharing detailed information about the changes, expected impact, and any guidance for users or other teams to adapt to the changes.

If needed, please also prepare PRs on documentation-website changes as part of the 3.0 release process.

Overall, this would help us deliver a smooth and seamless upgrade experience.

Thanks.

Changes in main not present on 2.x:

• Migrate to Apache HttpClient / Core 5.x #2166
• Support for HTTP/2 (server-side) #2051
• Set security plugin 3.0.0 baseline JDK version to JDK-21 #4457
• [RFC] Retire support for V6 configuration #4493
• Bump OpenSAML to 5.1.3 #5038
• SAML auth uses legacy _opendistro route #2060
• Remove deprecated settings from code #4117
• Protected indices are marked for deprecation--they should no longer be marked this way.  #3190
• transport_enabled setting on an auth domain and authorizer may be unnecessary after transport client removal  #3191

Major Updates

• [FEATURE] Deprecate the usage of ThreadContext.stashContext in plugins #2487 - Meta Issue
• [Proposal] Resource Permissions and Sharing #4500

[shikharj05]

@cwperks @peterzhuamazon what do you think about removing support for opendistro in APIs? For example APIs like -

https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/ssl/rest/SecuritySSLInfoAction.java#L52

security/src/main/java/org/opensearch/security/dlic/rest/support/Utils.java

Line 207 in 472f7c7

return addRoutesPrefix(routes, LEGACY_PLUGIN_API_ROUTE_PREFIX, PLUGIN_API_ROUTE_PREFIX);

Do we need to explicitly mark them deprecated before removing or we can remove them?

[cwperks]

@shikharj05 I support that idea, but #2060 is a prerequisite to that.

[cwperks]

Will there ever be plans to change the name of the security index (currently .opendistro_security?) I see a couple of options if it ever needs to be renamed:

1. Allow fresh clusters to start with the new index name (.opensearch_security?)
2. For clusters with an existing security index
   • Create an alias (.security)
   • First point the alias to .opendistro_security
   • Create a the new index (.opensearch_security).
   • Lock the index temporarily
   • Migrate data from .opendistro_security -> .opensearch_security
   • Flip the alias to point to .opensearch_security

All of that would be contingent on adding alias support for the security index.

There are still other instances in this repo of using legacy language like opendistro and kibana.

[shikharj05]

I like the idea of aliasing to security index in general. This could potentially also help with cases like - #5010

This could be very similar to what OSD does during upgrades/migrations.

[krishna-ggk]

The serialization improvement from 2.11 (#2780) was promising, but had to be disabled due to performance regressions with DLS.

security/src/main/java/org/opensearch/security/support/SerializationFormat.java

Line 28 in ec99e7e

public static SerializationFormat determineFormat(final Version version) {

The original issue also explored more alternatives.

@cwperks @DarshitChanpura @shikharj05 - do you think we can target this? (While technically this can be done in a minor version, a major version seems more appropriate to improvise on).

[krishna-ggk]

#5052 proposes a bunch of platform enhancements. The 4th one in that (Roles Injection Deprecation) doesn't seem to be covered under 3.0. A major version looks like the right time to explore this. Thoughts @cwperks ?

[shikharj05]

@shikharj05 I support that idea, but #2060 is a prerequisite to that.

I think we will need to explicitly mark these deprecated and document them for deprecation in 2.19. Post that, we can take a decision if we want to deprecate from 3.0. If we think the timelines are too close, we can move opendistro deprecation to next major version.

For e.g. the SAML path change should be documented in places like here for users.

https://semver.org/#how-should-i-handle-deprecating-functionality

Generally, SAML path changes might be more involved - changing recipient and assertion fields require migration support to not break upgrading clusters.

[cwperks]

#5052 proposes a bunch of platform enhancements. The 4th one in that (Roles Injection Deprecation) doesn't seem to be covered under 3.0. A major version looks like the right time to explore this. Thoughts @cwperks ?

A major version upgrade would be the right time to remove InjectSecurity from common-utils, but I don't think 3.0.0 is a feasible time frame. Ideally, I'd like to make the replacement available for fresh clusters behind a feature flag and have the option available to use the new model for scheduled job security.

For existing plugins that rely on InjectSecurity there needs to be enough notice to create a migration process from the existing model -> new model where security stores the information necessary for authorization at job runtime.

A few high-level steps to complete Roles Injection Deprecation and Replacement:

1. Add necessary hooks that allow the security plugin to inject information necessary for authz
2. Add a mechanism in the security plugin to store necessary information (i.e. the creator of the job, could either be a user or API Token)
3. Implement the hooks from 1) in the security plugin to retrieve information from 2) and inject it into the context

For plugins that use the existing InjectSecurity model, there would need to be a migration to extract information about user and roles currently stored in the jobs index and migrate them to the central store of the security plugin.

One option would be something akin to the Migrate API to perform this migration. Plugins that currently use InjectSecurity don't have a uniform way of storing user and roles information, but I think its reasonable for security to be conscious of this fact and implement the migration accordingly. Of course, it would only limit this to plugins in the default distribution.

The most common way user info is stored in jobs indices is in the common-utils User.toXContent() format. Example of GET Detector from AD: https://opensearch.org/docs/latest/observing-your-data/ad/api/#get-detector

Different plugins store this in different fields of the job's metadata. I have seen user and owner, but there may be others as well. There may also be different formats for storing the user info as plugins primarily store the whole object only to reference the backend_roles.

I'd love to get 1 and 2 from #5052 into 3.0.0, but 3 and 4 would be a stretch. As you pointed out, the end goal of #5052 would be to remove all security related code from common-utils.

[cwperks]

The serialization improvement from 2.11 (#2780) was promising, but had to be disabled due to performance regressions with DLS.

security/src/main/java/org/opensearch/security/support/SerializationFormat.java

Line 28 in ec99e7e

public static SerializationFormat determineFormat(final Version version) {
The original issue also explored more alternatives.

@cwperks @DarshitChanpura @shikharj05 - do you think we can target this? (While technically this can be done in a minor version, a major version seems more appropriate to improvise on).

Part of #4380 is removing DLS/FLS/FieldMasking rules from the ThreadContext headers which was a major culprit for serde issues. See the PR description from #4706 for more details, effectively the data structure would contain a map of all concrete indices from the user's role mappings to the DLS/FLS/FieldMasking rules applicable to those indices regardless of what indices the current request targeted.

For the rollout of Optimized Privilege Evaluation in 2.19, the FLS/DLS/FieldMasking improvements were intentionally left out but are planned for inclusion in a subsequent release (like 3.0.0)

Edit: Nils has written 2 blogs (looks like a 3rd yet to be published on DLS/FLS/FieldMasking is particular) around the Optimized Privilege Evaluation

1. Part 1 - https://eliatra.com/blog/performance-improvements-for-the-access-control-layer-of-opensearch/
2. Part 2 - https://eliatra.com/blog/performance-improvements-for-the-access-control-layer-of-opensearch-2/