[Sessions] Reindexing the .kibana_security_session_1 index to the 8.x format.

[SiddharthMantri]

Closes #200603

Summary

Reindexes the Kibana Security session system index to the 8.x format to support 9.0 readiness.

Release note

Creates Kibana Security session index to only if the kibana_security_session_1 index or the reindexed version do not exist.

Notes

How to test

For this test, you'll need at least 3 copies of Kibana cloned locally. One each on 7.17, 8.x and main - ensuring you've run yarn kbn bootstrap on each of them.

Step 0. Verify on the PR branch

• Start ES as yarn es snapshot --license=trial
• Start kibana yarn start --no-base-path
• Login to kibana in a private browsing window
• Navigate to dev tools and run

GET .kibana_security_session/_alias

• You should see

{
  ".kibana_security_session_1": {
    "aliases": {
      ".kibana_security_session": {
        "is_write_index": true,
        "is_hidden": true
      }
    }
  }
}

This indicates that there were no aliases/index present and the new index was created.

Step 1. On 7.17

• Run ES with yarn es snapshot --license=trial -E path.data=/tmp/esdata
• Run kibana
• Login with the elastic user
• Navigate to dev tools and run the following query

GET .kibana_security_session_1/_search
{
  "query": {
    "match_all": {}
  }
}

• You should see your current session being returned as the result for this query
• You can now shut down ES and kibana.

Step 2. On 8.x

• Run ES with yarn es snapshot --license=trial -E path.data=/tmp/esdata <--- point to the same folder as the previous run
• Run kibana, open a private browser window and login.
• Navigate to Kibana upgrade assistant and Migrate system indices and wait for it to run.
• Now in Dev tools, run the same query. You should see two sessions.
  • One with the idleSessionTimeout returned as null and the other one containing a value - indicating one was created on 7.x and the other in 8.x
• Make a backup of the data folder cp -r /tmp/esdata /tmp/esdatabkp

Step 3(OPTIONAL). On main (without the changes in this PR)

• Run ES with yarn es snapshot --license=trial -E path.data=/tmp/esdata
• This should throw an error

Step 4. On 8.x

• First use the backup for the path cp -r /tmp/esdatabkp /tmp/esdata2
• Start ES only (do not run Kibana yet) by pointing to the copy: yarn es snapshot --license=trial -E path.data=/tmp/esdata2
• ES should start up and you need to delete 1 index and 2 datastreams using the ES APIs and any method you prefer. For your convenience, you can use the same script as mine:

import axios from 'axios';

const clearIndexAndDatastream = async () => {
  {
    const res = await axios.delete(
      "http://localhost:9200/.kibana-event-log-7.17.28-000001",
      {
        headers: {
          Authorization: "Basic ZWxhc3RpYzpjaGFuZ2VtZQ==",
          accept: "*/*",
          "Content-Type": "application/json",
          "Kbn-Xsrf": "true",
        },
      }
    );
    console.log("deleted index:", JSON.stringify(res.data));
  }

  {
    const res = await axios.delete(
      "http://localhost:9200/_data_stream/ilm-history-5",
      {
        headers: {
          Authorization: "Basic ZWxhc3RpYzpjaGFuZ2VtZQ==",
          accept: "*/*",
          "Content-Type": "application/json",
          "Kbn-Xsrf": "true",
        },
      }
    );
    console.log("deleted ds1:", JSON.stringify(res.data));
  }
  {
    const res = await axios.delete(
      "http://localhost:9200/_data_stream/.logs-deprecation.elasticsearch-default",
      {
        headers: {
          Authorization: "Basic ZWxhc3RpYzpjaGFuZ2VtZQ==",
          accept: "*/*",
          "Content-Type": "application/json",
          "Kbn-Xsrf": "true",
        },
      }
    );
    console.log("deleted ds2:", JSON.stringify(res.data));
  }
};

clearIndexAndDatastream();

You should see the result as:

deleted index: {"acknowledged":true}
deleted ds1: {"acknowledged":true}
deleted ds2: {"acknowledged":true}

• Now login to Kibana in a private browsing window and navigate to Upgrade assistant and run the migration.
• Navigating to devtools and running the same query as above will show you three results. One with no idleTimeout and 2 with timeouts (One on 7.x and two on 8.x format respectively)

GET .kibana_security_session_1/_search
{
  "query": {
    "match_all": {}
  }
}

• You can now shut ES and kibana at this point.

Step 5. On the branch of this PR

• Run ES with yarn es snapshot --license=trial -E path.data=/tmp/esdata2
• Run Kibana and login using a private window.
• Navigating to dev tools and run:

GET .kibana_security_session/_alias

To show a result as:

{
  ".kibana_security_session_1-reindexed-for-9": {
    "aliases": {
      ".kibana_security_session": {
        "is_hidden": true
      },
      ".kibana_security_session_1": {
        "is_hidden": true
      }
    }
  }
}

This indicates that no new index was created and we are using the reindexed version from 8.x.

• You should also run the query to check for sessions:

GET .kibana_security_session_1/_search
{
  "query": {
    "match_all": {}
  }
}

• This should return 4 sessions in the results

This confirms that the session was re-indexed correctly using the right aliases.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Unit or functional tests were updated or added to match the most common scenarios
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

• See some risk examples
• ...

[SiddharthMantri]

@elasticmachine merge upstream

[SiddharthMantri]

@elasticmachine merge upstream

[SiddharthMantri]

/ci

[SiddharthMantri]

@elasticmachine merge upstream

[SiddharthMantri]

/ci

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: a528735

Metrics [docs]

✅ unchanged

History

• 💔 Build #264732 failed e27b65f
• 💔 Build #263134 failed d750edf
• 💔 Build #263028 failed 286df83

[jeramysoucy]

LGTM! Thanks for your patience and help with this one, Sid! I tested following your updated instructions (thank for that!). To confirm the issue and the fix, after deleting the problematic event log index and two data streams and running the index migration in UA from the 8.x branch, I copied the es data twice - once to run with main, and the other to run with your PR. This allowed me to run from both main and your PR without additional errors and see the difference. From main we get the following (expected) error when starting Kibana:

illegal_argument_exception: The provided expression [.kibana_security_session_1] matches an alias, specify the corresponding concrete indices instead.
[2025-01-21T10:30:27.820+01:00][ERROR][plugins.security.session.index] Failed to attach alias to session index: illegal_argument_exception

From your PR, all goes well and I can log in and see the total session count reflected accurately. 🚀

[SiddharthMantri]

@elasticmachine merge upstream

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12908815730

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation