Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[elasticsearch] Disable transform validation in system tests #12840

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

[elasticsearch] Disable transform validation in system tests #12840

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d35ac08
Disable transform validation in system tests
mrodm Feb 19, 2025
466d68d
Add comment
mrodm Feb 19, 2025
f9bc53b
Test with elastic-package - main f61be1a397ac
mrodm Feb 19, 2025
ed5fecb
Revert "Test with elastic-package - main f61be1a397ac"
mrodm Feb 19, 2025
9895eba
Updated comment
mrodm Feb 19, 2025
af4eb8e
Update comment about skip_transform_validation
mrodm Feb 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions packages/elasticsearch/data_stream/index/_dev/test/system/test-default-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,6 @@ vars:
username: elastic
password: changeme
data_stream: ~
# Currently, when trying to run the preview of the transform logs-elasticsearch.index_pivot-default-0.1.0, fails with this error:
# [400 Bad Request] {"error":{"root_cause":[{"type":"status_exception","reason":"Source indices have been deleted or closed."}],"type":"status_exception","reason":"Source indices have been deleted or closed."},"status":400}
skip_transform_validation: true
Copy link
Contributor Author

@mrodm mrodm Feb 19, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When testing with a newer stack version (tested with 8.17.1, 8.17.2 and 8.18.0-SNAPSHOT), there are no errors.

However, testing with the current stack version set in the manifest (8.10.1), this is failing with the error mentioned in the comment

  • Maybe is this caused due to the destination index is not created?

failed to preview transform "logs-elasticsearch.index_pivot-default-0.2.0": [400 Bad Request] {"error":{"root_cause":[{"type":"status_exception","reason":"Source indices have been deleted or closed."}],"type":"status_exception","reason":"Source indices have been deleted or closed."},"status":400}

Using other Elastic stack versions 8.14.0, 8.16.0 and 8.17.0, the error is different:

  • Maybe the filters/query in the transform cause that there are no documents to be processed?

failed to preview transform "logs-elasticsearch.index_pivot-default-0.2.0": [400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"must specify at least one document in [docs]"}],"type":"illegal_argument_exception","reason":"must specify at least one document in [docs]"},"status":400}

Copy link
Contributor Author

@mrodm mrodm Feb 19, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just found in the documentation, that it looks like it is expected that this transform just works in 8.17.1 and onward

integrations/packages/elasticsearch/docs/README.md

Line 2662 in 1a820b6

For version 8.17.1+ of the module and collected data, the integration also installs a transform job called `logs-elasticsearch.index_pivot-default-{VERSION}`. This transform **isn't started by default** (Stack management > Transforms), but will perform the following once activated:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would it be better here ? @jsoriano

I was thinking to rephrase the comment set here (and keep disabled the transform validation):

# Current `index_pivot` transform is expected to work just on stack versions >=8.17.1
# remove this setting once the kibana constraint in the package manifest is >=8.17.1
skip_transform_validation: true

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, lets keep this validation disabled by now if it is going to be problematic.

3 changes: 3 additions & 0 deletions packages/elasticsearch/data_stream/index_recovery/_dev/test/system/test-default-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,6 @@ vars:
data_stream:
vars:
active.only: false
# Currently, when trying to run the preview of the transform logs-elasticsearch.index_pivot-default-0.2.0, fails with this error:
# [400 Bad Request] {"error":{"root_cause":[{"type":"status_exception","reason":"Source indices have been deleted or closed."}],"type":"status_exception","reason":"Source indices have been deleted or closed."},"status":400}
skip_transform_validation: true
3 changes: 3 additions & 0 deletions packages/elasticsearch/data_stream/index_summary/_dev/test/system/test-default-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,6 @@ vars:
username: elastic
password: changeme
data_stream: ~
# Currently, when trying to run the preview of the transform logs-elasticsearch.index_pivot-default-0.1.0, fails with this error:
# [400 Bad Request] {"error":{"root_cause":[{"type":"status_exception","reason":"Source indices have been deleted or closed."}],"type":"status_exception","reason":"Source indices have been deleted or closed."},"status":400}
skip_transform_validation: true