[ML][Fleet] Update Transform installation mechanism to support upgrad…

…e paths (#142920)

## Summary

This PR:
- Adds index aliases to the destination index for Transforms:
- It will automatically appends `{{package-version}}` name to the
`destination_index_name` specified in transform.yml
- Create a `{destination_index_name}.all` that points to all the
destination indices from all the previous versions and new version
- Create a `{destination_index_name}.latest` that points to just the
destination index of the new version
- Upgrading package to a new version no longer deletes the destination
index
- Downgrading package to an older version (e.g. from v3 to v1) will:
- Delete the transform from the newer version (v3), create transform for
the older version (v1)
- If the older version was previously installed and the destination
index of the older version v1 already exists: update the alias
`{destination_index_name}.latest` to point to destination index v1.
- If the older version was never installed and destination index of the
older version v1 does not exist: create the destination index with
`{destination_index_name}.all` and `{destination_index_name}.latest`
alias.

- Support installing transforms concurrently and sequentially. 
- If the `order` is specified in the `transform.yml`'s `_meta` section,
and all the numerical order are unique, transforms will be created and
started sequentially. If not, they will be created and started
concurrently.
- Support versioning of transforms. If `fleet_transform_version` is
specified in `transform.yml`'s `_meta` section:
- If `fleet_transform_version` changed (either incremented or
decremented): delete old transform, keep the old destination index,
install new index templates, component templates, and transform
- If `fleet_transform_version` remains the same: keep old transform,
keep the old destination index, do nothing new
- Fixes an issue with the mappings and template not being applied to the
destination index correctly when the destination index has an ingest
pipeline. Previously, when the transform is associated with an ingest
pipeline, we add the ingest pipeline to the settings when calling `PUT
index/{transform-destination-index}`. This in turns makes the settings
and mappings from the component templates not apply correctly to the
destination. This PR changes so that it will add the pipeline to the
component template.

Technical changes:
- [Adds a new `ElasticsearchAssetType` for `index`

](https://github.com/elastic/kibana/pull/142920/files#diff-395b753abcf65cdc07993651d6211a49194a76c0497e5f234ea13736cf24a2c0)
- [Adds a new `version` for `PACKAGES_SAVED_OBJECT_TYPE`

](https://github.com/elastic/kibana/pull/142920/files#diff-4e164e3802d5171bf96a2cf9c91c20e97c5e0b74b2f93187a072d9a3139f1c18)


### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)


### Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to
identify risks that should be tested prior to the change/feature
release.

When forming the risk matrix, consider some of the following examples
and how they may potentially impact the change:

| Risk | Probability | Severity | Mitigation/Notes |

|---------------------------|-------------|----------|-------------------------|
| Multiple Spaces—unexpected behavior in non-default Kibana Space.
| Low | High | Integration tests will verify that all features are still
supported in non-default Kibana Space and when user switches between
spaces. |
| Multiple nodes—Elasticsearch polling might have race conditions
when multiple Kibana nodes are polling for the same tasks. | High | Low
| Tasks are idempotent, so executing them multiple times will not result
in logical error, but will degrade performance. To test for this case we
add plenty of unit tests around this logic and document manual testing
procedure. |
| Code should gracefully handle cases when feature X or plugin Y are
disabled. | Medium | High | Unit tests will verify that any feature flag
or plugin combination still results in our service operational. |
| [See more potential risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) |


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

Co-authored-by: Kibana Machine <[email protected]>

src/core/server/integration_tests/saved_objects/migrations/check_registered_types.test.ts

@@ -85,7 +85,7 @@ describe('checking migration metadata changes on all registered SO types', () =>
         "endpoint:user-artifact": "f94c250a52b30d0a2d32635f8b4c5bdabd1e25c0",
         "endpoint:user-artifact-manifest": "8c14d49a385d5d1307d956aa743ec78de0b2be88",
         "enterprise_search_telemetry": "fafcc8318528d34f721c42d1270787c52565bad5",
-        "epm-packages": "fe3716a54188b3c71327fa060dd6780a674d3994",
+        "epm-packages": "2915aee4302d4b00472ed05c21f59b7d498b5206",
         "epm-packages-assets": "9fd3d6726ac77369249e9a973902c2cd615fc771",
         "event_loop_delays_daily": "d2ed39cf669577d90921c176499908b4943fb7bd",
         "exception-list": "fe8cc004fd2742177cdb9300f4a67689463faf9c",

x-pack/plugins/fleet/common/types/models/epm.ts

@@ -101,6 +101,7 @@ export enum KibanaSavedObjectType {
 }
 
 export enum ElasticsearchAssetType {
+  index = 'index',
   componentTemplate = 'component_template',
   ingestPipeline = 'ingest_pipeline',
   indexTemplate = 'index_template',
@@ -109,6 +110,10 @@ export enum ElasticsearchAssetType {
   dataStreamIlmPolicy = 'data_stream_ilm_policy',
   mlModel = 'ml_model',
 }
+export type FleetElasticsearchAssetType = Exclude<
+  ElasticsearchAssetType,
+  ElasticsearchAssetType.index
+>;
 
 export type DataType = typeof dataTypes;
 export type MonitoringType = typeof monitoringTypes;
@@ -313,7 +318,7 @@ export type ElasticsearchAssetParts = AssetParts & {
 
 export type KibanaAssetTypeToParts = Record<KibanaAssetType, KibanaAssetParts[]>;
 export type ElasticsearchAssetTypeToParts = Record<
-  ElasticsearchAssetType,
+  FleetElasticsearchAssetType,
   ElasticsearchAssetParts[]
 >;
 

x-pack/plugins/fleet/public/applications/integrations/sections/epm/constants.tsx

@@ -35,6 +35,9 @@ export const AssetTitleMap: Record<DisplayedAssetType, string> = {
   transform: i18n.translate('xpack.fleet.epm.assetTitles.transforms', {
     defaultMessage: 'Transforms',
   }),
+  index: i18n.translate('xpack.fleet.epm.assetTitles.indices', {
+    defaultMessage: 'Indices',
+  }),
   index_pattern: i18n.translate('xpack.fleet.epm.assetTitles.indexPatterns', {
     defaultMessage: 'Index patterns',
   }),

x-pack/plugins/fleet/server/saved_objects/index.ts

@@ -262,6 +262,7 @@ const getSavedObjectTypes = (
           properties: {
             id: { type: 'keyword' },
             type: { type: 'keyword' },
+            version: { type: 'keyword' },
           },
         },
         installed_kibana: {

x-pack/plugins/fleet/server/services/epm/elasticsearch/index/index.ts

@@ -0,0 +1,8 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+export { updateIndexSettings } from './update_settings';

x-pack/plugins/fleet/server/services/epm/elasticsearch/index/update_settings.ts

@@ -0,0 +1,31 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import type { ElasticsearchClient } from '@kbn/core-elasticsearch-server';
+
+import type { IndicesIndexSettings } from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
+
+import { retryTransientEsErrors } from '../retry';
+
+export async function updateIndexSettings(
+  esClient: ElasticsearchClient,
+  index: string,
+  settings: IndicesIndexSettings
+): Promise<void> {
+  if (index) {
+    try {
+      await retryTransientEsErrors(() =>
+        esClient.indices.putSettings({
+          index,
+          body: settings,
+        })
+      );
+    } catch (err) {
+      throw new Error(`could not update index settings for ${index}`);
+    }
+  }
+}

x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/common.ts

@@ -6,3 +6,8 @@
  */
 
 export { getAsset } from '../../archive';
+
+// Index alias that points to just one destination index from the latest package version
+export const TRANSFORM_DEST_IDX_ALIAS_LATEST_SFX = '.latest';
+// Index alias that points to all of the destination indices from all the package versions
+export const TRANSFORM_DEST_IDX_ALIAS_ALL_SFX = '.all';