Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable running kola AWS tests using separate credentials #130

Merged
merged 4 commits into from
Sep 12, 2019
Merged

Enable running kola AWS tests using separate credentials #130

merged 4 commits into from
Sep 12, 2019

Conversation

dustymabe
Copy link
Member

We'd like to be able to run our kola AWS tests in a separate
account than the prod Fedora AWS account. The following
collection of commits enables us to specify different creds for
our build uploads and our kola tests.

Commit title summary:

  • Add new AWS creds secret for running kola tests
  • Move current AWS_CONFIG_FILE var to AWS_FCOS_BUILDS_BOT_CONFIG
  • Jenkinsfile: remove AWS_SDK_LOAD_CONFIG var
  • Jenkinsfile: add context comments to Kola Runs stage

@dustymabe
Copy link
Member Author

This builds on #128.

cc @jlebon @arithx - this should get us all set up and close the book on AWS replication and also AWS kola tests.

@dustymabe dustymabe mentioned this pull request Sep 11, 2019
Merged
@arithx
Copy link
Contributor

arithx commented Sep 11, 2019

Any thoughts on using a different profile rather than a secondary credential file? It would allow us to not have to specify the export in every function context while still using the community account in kola (via the --aws-profile flag).

@dustymabe
Copy link
Member Author

Any thoughts on using a different profile rather than a secondary credential file? It would allow us to not have to specify the export in every function context while still using the community account in kola (via the --aws-profile flag).

I thought about something similar, which is just exporting the builds-bot config as AWS_CONFIG_FILE in the pod.yaml and then overriding that when we call the kola for AWS testing. I decided to go with explicitly setting the variable everywhere. I think it helps us identify where we're calling into AWS (primarily a bunch of S3 operations), but I'm not super tied to it.

There are 3 options:

  • Use two configs and two corresponding env vars (the current status of this PR)
  • Use two configs and set the builds bot one as the default AWS_CONFIG_FILE var. Override AWS_CONFIG_FILE with AWS_FCOS_KOLA_BOT_CONFIG right before call to kola.
  • Use one config with profiles where the default profile is the builds bot config and the kola profile is the kola bot config and then override with the --aws-profile flag.

@cgwalters
Copy link
Member

We could also use a separate pod for testing, without the prod creds even mounted in.

@cgwalters
Copy link
Member

(I'd like to make more use of the fact that we're using Kubernetes, it's easy to run separated containers, rather than the "one big pod doing everything" flow we have today)

@arithx
Copy link
Contributor

arithx commented Sep 11, 2019

I don't have a personal preference for any of the 3 options. Just wanted to throw out the idea to make sure that it was also considered.

jlebon
jlebon reviewed Sep 11, 2019
View reviewed changes
HACKING.md
aws_secret_access_key=key
EOF

oc create secret generic aws-fcos-kola-bot-config --from-file=config=/path/to/kola-secret
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the IAM account called fcos-kola-bot? I'd like to keep the exact name of the IAM in the secret name to make things easier to track.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes the IAM user in the fedora community account is called fcos-kola-bot.

So no need for changes?

@jlebon
Copy link
Member

jlebon commented Sep 11, 2019

We could also use a separate pod for testing, without the prod creds even mounted in.

Yeah, I think we should do this eventually (I'd like to even make the building and uploading bits separate to narrow it down further -- related: #7).

For reference, here's an approach where we edit the pod template on-the-fly based on some parameters: https://github.com/coreos/coreos-ci-lib/blob/67ec2b13db811ea9c601b210bd52aba51aa2791a/vars/coreos.groovy.

Anyway, I don't think we should block on this, but should probably open a ticket (or we can just re-use #7 for it).

jlebon
jlebon approved these changes Sep 11, 2019
View reviewed changes
Copy link
Member

@jlebon jlebon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@dustymabe
Copy link
Member Author

rebased on top of new changes from #128 - will give this another test and then merge both #128 and this PR

@dustymabe dustymabe mentioned this pull request Sep 12, 2019
Merged
@dustymabe dustymabe marked this pull request as ready for review September 12, 2019 15:54
@dustymabe
Jenkinsfile: add context comments to Kola Runs stage
80fcc09 
These help us understand why we chose the jnlp container
and also why we don't block on the kola AWS runs.
@dustymabe
Jenkinsfile: remove AWS_SDK_LOAD_CONFIG var
cb4a8a1 
Setting this is no longer needed after [1] was merged into the
mantle codebase.

[1] coreos/mantle#1028
@dustymabe
Move current AWS_CONFIG_FILE var to AWS_FCOS_BUILDS_BOT_CONFIG
244bfd0 
This is prep to allow for us to have more than one AWS config. One
for uploading/replicating/releasing builds and another one for testing
those builds using kola. For now we'll use the AWS_FCOS_BUILDS_BOT_AWS_CONFIG
var and explicitly set AWS_CONFIG_FILE each time we make a call that
will use the AWS config for the fcos-builds-bot user in the prod Fedora
AWS account.
@dustymabe
Add new AWS creds secret for running kola tests
86557f5 
In this PR we add the aws-fcos-kola-bot-config secret that
represents credentials to use when running kola tests in AWS.

This will allow us to upload builds to the prod Fedora Account
(more locked down security) and do testing in our community Fedora
Account where we have control over the credentials handed out.
@dustymabe dustymabe merged commit cfbd6ae into coreos:master Sep 12, 2019
@dustymabe dustymabe deleted the kola-creds branch September 12, 2019 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jlebon jlebon jlebon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dustymabe @arithx @cgwalters @jlebon