Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 Inventory #2074

Merged

Conversation

abraverm
Copy link
Contributor

@abraverm abraverm commented May 2, 2024

SUMMARY

Add S3 inventory support - https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html

ISSUE TYPE
  • Feature Pull Request
COMPONENT NAME

S3 Bucket

Copy link

github-actions bot commented May 2, 2024

Docs Build 📝

Thank you for contribution!✨

This PR has been merged and your docs changes will be incorporated when they are next published.

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/97252baad8174e92b013177b324d7c63

✔️ ansible-galaxy-importer SUCCESS in 5m 16s
✔️ build-ansible-collection SUCCESS in 16m 06s
✔️ ansible-test-splitter SUCCESS in 6m 46s
integration-amazon.aws-1 FAILURE in 7m 48s
Skipped 43 jobs

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/78c221ae076b41e5a75d7118bf5d89ad

✔️ ansible-galaxy-importer SUCCESS in 5m 31s
✔️ build-ansible-collection SUCCESS in 15m 09s
✔️ ansible-test-splitter SUCCESS in 6m 47s
integration-amazon.aws-1 FAILURE in 9m 20s
Skipped 43 jobs

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/334fbed078cc4145bd013b2dcb3c60ff

✔️ ansible-galaxy-importer SUCCESS in 5m 50s
✔️ build-ansible-collection SUCCESS in 15m 19s
✔️ ansible-test-splitter SUCCESS in 5m 27s
integration-amazon.aws-1 FAILURE in 9m 13s
Skipped 43 jobs

@abraverm
Copy link
Contributor Author

abraverm commented May 2, 2024

CI failed because of permissions, I have created a PR to add them - ansible/aws-ci-admin#6

@gravesm
Copy link
Member

gravesm commented May 3, 2024

recheck

Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/d471314f5dc8436dbce3626b7218b9ad

✔️ ansible-galaxy-importer SUCCESS in 5m 02s
✔️ build-ansible-collection SUCCESS in 16m 21s
✔️ ansible-test-splitter SUCCESS in 6m 20s
✔️ integration-amazon.aws-1 SUCCESS in 6m 06s
Skipped 43 jobs

Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/138400da26b94a99a5ebca79504a8072

✔️ ansible-galaxy-importer SUCCESS in 5m 13s
✔️ build-ansible-collection SUCCESS in 15m 20s
✔️ ansible-test-splitter SUCCESS in 6m 19s
✔️ integration-amazon.aws-1 SUCCESS in 7m 41s
Skipped 43 jobs

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/46eb11b0c8e74784b42b7a6fd9db886c

⚠️ ansible-galaxy-importer SKIPPED Skipped due to failed job build-ansible-collection
build-ansible-collection RETRY_LIMIT in 6m 35s
✔️ ansible-test-splitter SUCCESS in 9m 10s
⚠️ integration-amazon.aws-1 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-2 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-3 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-4 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-5 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-6 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-7 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-8 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-9 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-10 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-11 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-12 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-13 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-14 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-15 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-16 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-17 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-18 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-19 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-20 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-21 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-amazon.aws-22 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-1 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-2 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-3 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-4 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-5 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-6 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-7 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-8 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-9 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-10 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-11 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-12 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-13 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-14 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-15 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-16 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-17 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-18 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-19 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-20 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-21 SKIPPED Skipped due to failed job build-ansible-collection
⚠️ integration-community.aws-22 SKIPPED Skipped due to failed job build-ansible-collection

@abraverm
Copy link
Contributor Author

abraverm commented Jun 6, 2024

recheck

Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/385d3bdfc40644f28d529066f2f79f9f

✔️ ansible-galaxy-importer SUCCESS in 6m 07s
✔️ build-ansible-collection SUCCESS in 16m 20s
✔️ ansible-test-splitter SUCCESS in 6m 09s
✔️ integration-amazon.aws-1 SUCCESS in 7m 21s
Skipped 43 jobs

@hakbailey hakbailey added this to the 8.1.0 milestone Jun 26, 2024
Copy link
Contributor

@hakbailey hakbailey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that multiple inventory configurations can be applied to a bucket, I would suggest this be updated to allow providing a list of configuration dicts. The way this is written currently, it only allows one configuration and only compares the provided configuration to the first result from list_bucket_inventory_configurations, which seems like it could cause unexpected problems.

Copy link
Contributor

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/ansible-collections/amazon.aws for 2074,82a3762d9e7c76dc471287fe018af6b9a441e4ae

@tremble tremble changed the title Inventory S3 Inventory Jul 18, 2024
@alinabuzachis alinabuzachis modified the milestones: 8.1.0, 8.2.0 Jul 18, 2024
Copy link
Contributor

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/ansible-collections/amazon.aws for 2074,ba0d246d20d5c4854b514756e959631f66cfc96b

Copy link
Contributor

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/ansible-collections/amazon.aws for 2074,3ec9bd3076b2116688f35e3f63d0fc3ff533028e

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/0b15c986bb6849448fb0c8ff70641f16

ansible-galaxy-importer FAILURE in 4m 36s
✔️ build-ansible-collection SUCCESS in 12m 51s
✔️ ansible-test-splitter SUCCESS in 4m 54s
integration-amazon.aws-1 FAILURE in 5m 31s
Skipped 43 jobs

@abraverm
Copy link
Contributor Author

recheck

Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/513ff4b1a3b240fda4511a05cdf9cc15

✔️ ansible-galaxy-importer SUCCESS in 4m 11s
✔️ build-ansible-collection SUCCESS in 12m 29s
✔️ ansible-test-splitter SUCCESS in 4m 56s
✔️ integration-amazon.aws-1 SUCCESS in 9m 42s
Skipped 43 jobs

Copy link
Contributor

@hakbailey hakbailey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abraverm Thank you for submitting these updates! In addition to the inline comments below, could you please add a changelog file and fix the linter errors by running the tox formatters with tox -m format? Let me know if you have questions.

plugins/modules/s3_bucket.py Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Show resolved Hide resolved
@abraverm
Copy link
Contributor Author

recheck

@abraverm
Copy link
Contributor Author

The tests are failing on things unrelated to this change:

ERROR: plugins/modules/lambda_info.py:51:73: W291: trailing whitespace
ERROR: plugins/modules/rds_instance.py:539:71: W291: trailing whitespace

Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/d50c37c5e2824cf2b07ad4d9c46a362a

✔️ ansible-galaxy-importer SUCCESS in 5m 01s
✔️ build-ansible-collection SUCCESS in 10m 30s
✔️ ansible-test-splitter SUCCESS in 4m 15s
✔️ integration-amazon.aws-1 SUCCESS in 11m 37s
✔️ integration-amazon.aws-2 SUCCESS in 7m 48s
Skipped 42 jobs

Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/67b59d0a10424e57b2ba02530a608848

✔️ ansible-galaxy-importer SUCCESS in 3m 24s
✔️ build-ansible-collection SUCCESS in 11m 48s
✔️ ansible-test-splitter SUCCESS in 4m 45s
✔️ integration-amazon.aws-1 SUCCESS in 15m 23s
✔️ integration-amazon.aws-2 SUCCESS in 9m 19s
Skipped 42 jobs

Copy link
Contributor

@tremble tremble left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking the time to submit this PR.

One significant change needed here, please do not set default values for net-new options. Two reasons:

  1. This will cause the default behaviour to go from not touching the inventory configuration, to wiping the inventory configuration
  2. The S3 modules also support non-AWS S3-like APIs, by setting a default this instantly breaks any of them where the API doesn't support the new feature. This is a bit of a hassle, but is what it is.

plugins/module_utils/s3.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/module_utils/s3.py Outdated Show resolved Hide resolved
plugins/module_utils/s3.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
@tremble tremble added the backport-8 PR should be backported to the stable-8 branch label Aug 28, 2024
@abraverm abraverm force-pushed the inventory branch 2 times, most recently from 1a7f384 to 9a7fa17 Compare August 29, 2024 20:51
Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/c1bca846ba5048e0936fffcc2c233985

✔️ ansible-galaxy-importer SUCCESS in 4m 37s
✔️ build-ansible-collection SUCCESS in 11m 06s
✔️ ansible-test-splitter SUCCESS in 4m 19s
✔️ integration-amazon.aws-1 SUCCESS in 11m 48s
integration-amazon.aws-2 FAILURE in 11m 15s
Skipped 42 jobs

Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/055cab44315247e68f8cde396d10b462

✔️ ansible-galaxy-importer SUCCESS in 4m 38s
✔️ build-ansible-collection SUCCESS in 10m 38s
✔️ ansible-test-splitter SUCCESS in 4m 18s
✔️ integration-amazon.aws-1 SUCCESS in 13m 42s
✔️ integration-amazon.aws-2 SUCCESS in 7m 51s
Skipped 42 jobs

@abraverm
Copy link
Contributor Author

@hakbailey , Github says that you have requested changes, but I can't find any - what am I missing?

@tremble
Copy link
Contributor

tremble commented Aug 30, 2024

@abraverm They're been "resolved" which is why they're not showing up. I'll request a fresh review from her.

@tremble tremble requested a review from hakbailey August 30, 2024 09:04
@tremble tremble added the mergeit Merge the PR (SoftwareFactory) label Aug 30, 2024
@tremble tremble added mergeit Merge the PR (SoftwareFactory) and removed mergeit Merge the PR (SoftwareFactory) labels Aug 30, 2024
Copy link
Contributor

Build succeeded (gate pipeline).
https://ansible.softwarefactory-project.io/zuul/buildset/459db7d5fb6e49c1a995b0cea97429ef

✔️ ansible-galaxy-importer SUCCESS in 5m 33s
✔️ build-ansible-collection SUCCESS in 10m 41s
✔️ ansible-test-splitter SUCCESS in 4m 24s
✔️ integration-amazon.aws-1 SUCCESS in 12m 11s
✔️ integration-amazon.aws-2 SUCCESS in 7m 26s
Skipped 42 jobs

@softwarefactory-project-zuul softwarefactory-project-zuul bot merged commit c96d250 into ansible-collections:main Aug 30, 2024
39 checks passed
Copy link

patchback bot commented Aug 30, 2024

Backport to stable-8: 💚 backport PR created

✅ Backport PR branch: patchback/backports/stable-8/c96d250169247c004ed36e205f9ff7c39f5da5b3/pr-2074

Backported as #2273

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback bot pushed a commit that referenced this pull request Aug 30, 2024
SUMMARY
Add S3 inventory support - https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html
ISSUE TYPE

Feature Pull Request

COMPONENT NAME
S3 Bucket

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Alexander Braverman Masis <[email protected]>
Reviewed-by: Mark Chappell
Reviewed-by: Alina Buzachis
(cherry picked from commit c96d250)
softwarefactory-project-zuul bot pushed a commit that referenced this pull request Aug 30, 2024
This is a backport of PR #2074 as merged into main (c96d250).
SUMMARY
Add S3 inventory support - https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html
ISSUE TYPE

Feature Pull Request

COMPONENT NAME
S3 Bucket

Reviewed-by: Mark Chappell
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8 PR should be backported to the stable-8 branch mergeit Merge the PR (SoftwareFactory)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants