-
Notifications
You must be signed in to change notification settings - Fork 170
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Release hash script and update hashes #230
Closed
Closed
Changes from 7 commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
03b69b7
Update Dockerfile with template for breaking build cache
ruffsl 8e4a537
Use digest to check InRelease
ruffsl 8942103
Merge branch 'master' into release_hash_shane
sloretz f96ab24
Add script to update InRelease hashes
sloretz bb68188
Update dockerfile InRelease hashes
sloretz 8931512
Add script to create official-images PRs
sloretz a6fcc52
Fix variable name
sloretz 7fcdc99
Main block reads HUB_REPO instead of hard coding
sloretz 815e3ff
Delete update_cache_busting_hashes.py
sloretz File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,147 @@ | ||
| #!/usr/bin/env python3 | ||
|
|
||
| import datetime | ||
| import difflib | ||
| import git # gitpython | ||
| import github | ||
| import os | ||
| import shutil | ||
| import tempfile | ||
|
|
||
|
|
||
| # Path to root of this repo | ||
| BASE_PATH = os.path.dirname(os.path.abspath(__file__)) | ||
| # Path to official docker library repo | ||
| DOCKER_LIBRARY_REPO_URL = '[email protected]:docker-library/official-images.git' | ||
| DOCKER_LIBRARY_REPO = 'docker-library/official-images' | ||
| # Path to fork to update with docker images | ||
| FORK_LIBRARY_REPO_URL = '[email protected]:ros-infrastructure/official-images.git' | ||
| FORK_LIBRARY_REPO_USER = 'ros-infrastructure' | ||
|
|
||
| # Token to use to interact with github | ||
| GIT_USER = os.environ.get('GITHUB_USER','') | ||
| GIT_EMAIL = os.environ.get('GITHUB_EMAIL','') | ||
| GIT_TOKEN = os.environ.get('GITHUB_TOKEN','') | ||
| GIT_AUTHOR = "{user} <{email}>".format(user=GIT_USER, email=GIT_EMAIL) | ||
| GIT_TOKEN = os.environ.get('GITHUB_TOKEN','') | ||
|
|
||
|
|
||
| PR_TEMPLATE = """ | ||
| # Update library definition for '{name}' | ||
|
|
||
| <details><summary>Diff</summary> | ||
|
|
||
| ```diff | ||
| {diff} | ||
| ``` | ||
| </details> | ||
| """ | ||
|
|
||
|
|
||
| def read_local_library_definition(name): | ||
| # Library definition file for `ros` is in `ros/ros` | ||
| with open(os.path.join(BASE_PATH, name, name), 'r') as fin: | ||
| return fin.read() | ||
|
|
||
|
|
||
| def read_upstream_library_definition(name): | ||
| # Get the manifest from the upstream repo | ||
| with tempfile.TemporaryDirectory(prefix='osrf-docker-images') as temp_dir: | ||
| git.Repo.clone_from(DOCKER_LIBRARY_REPO_URL, temp_dir, depth=1) | ||
| with open(os.path.join(temp_dir, 'library', name)) as fin: | ||
| return fin.read() | ||
|
|
||
|
|
||
| def pr_with_sentinel_exists(sentinel): | ||
| gh_handle = github.Github(GIT_TOKEN) | ||
| repo = gh_handle.get_repo(DOCKER_LIBRARY_REPO) | ||
| for pr in repo.get_pulls(state='open'): | ||
| if sentinel in pr.body: | ||
| return True | ||
| return False | ||
|
|
||
|
|
||
| def diff_library_definitions(local_ldf, upstream_ldf): | ||
| differ = difflib.Differ() | ||
| diff = differ.compare( | ||
| upstream_ldf.splitlines(keepends=True), | ||
| local_ldf.splitlines(keepends=True)) | ||
| diff = ''.join(diff) | ||
| return diff | ||
|
|
||
|
|
||
| def update_library_definition_fork(name): | ||
| # Create branch on fork that is up to date with upstream | ||
| # write changed manifest to fork | ||
| # commit to new branch | ||
| # push branch to fork | ||
| branch_name = None | ||
| with tempfile.TemporaryDirectory(prefix='osrf-docker-images') as temp_dir: | ||
| # Clone fork repo locally with a remote for the official one | ||
| repo = git.Repo.clone_from(FORK_LIBRARY_REPO_URL, temp_dir, depth=1) | ||
| remote = repo.create_remote('upstream', DOCKER_LIBRARY_REPO_URL) | ||
| if not remote.exists(): | ||
| raise RuntimeError('Failed to find official docker library') | ||
| remote.fetch() | ||
|
|
||
| # Create branch on fork repo tracking official docker library | ||
| d = datetime.datetime.utcnow() | ||
| d_str = d.strftime(r'%Y-%m-%d_%H-%M-%S') | ||
| branch_name = '{}/{}'.format(name, d_str) | ||
| head = repo.create_head(branch_name, remote.refs.master) | ||
| head.checkout() | ||
|
|
||
| # Update the library definition file on the fork | ||
| shutil.copy( | ||
| os.path.join(BASE_PATH, name, name), | ||
| os.path.join(temp_dir, 'library', name)) | ||
|
|
||
| repo.git.add(all=True) | ||
| message = 'Update library definition for ' + name | ||
| repo.git.commit(message=message, author=GIT_AUTHOR) | ||
| repo.remotes.origin.push(branch_name, set_upstream=True) | ||
|
|
||
| print('created branch', branch_name) | ||
|
|
||
| return branch_name | ||
|
|
||
|
|
||
| def create_upstream_pull_request(name, diff, branch_name): | ||
| # open pull request with template | ||
| gh_handle = github.Github(GIT_TOKEN) | ||
| repo = gh_handle.get_repo(DOCKER_LIBRARY_REPO) | ||
| pr = repo.create_pull( | ||
| title='Update ' + name, | ||
| head='{}:{}'.format(FORK_LIBRARY_REPO_USER, branch_name), | ||
| base='master', | ||
| body=PR_TEMPLATE.format(name=name, diff=diff)) | ||
| print('Created PR', pr.url) | ||
|
|
||
|
|
||
| def update_upstream_library_definition(name): | ||
| local_ldf = read_local_library_definition(name) | ||
| upstream_ldf = read_upstream_library_definition(name) | ||
|
|
||
| if local_ldf == upstream_ldf: | ||
| print('Upstream LDF and local LDF for {} are in sync'.format(name)) | ||
| return | ||
|
|
||
| diff = diff_library_definitions(local_ldf, upstream_ldf) | ||
| print('--- Diff for {}---'.format(name)) | ||
| print(diff) | ||
| print('--- End diff ---') | ||
|
|
||
| if pr_with_sentinel_exists(diff): | ||
| print('Upstream PR alredy created, ignoring') | ||
| return | ||
|
|
||
| fork_branch = update_library_definition_fork(name) | ||
|
|
||
| # Create pull request | ||
| create_upstream_pull_request(name, diff, fork_branch) | ||
|
|
||
|
|
||
| if __name__ == '__main__': | ||
|
|
||
| update_upstream_library_definition('ros') | ||
sloretz marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| update_upstream_library_definition('gazebo') | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this pushing the manifest (post merged) copied from osrf/docker_images to the
FORK_LIBRARY_REPO_URLor only that from the checked out PR? To consolidate all the release changes for a given docker repo to one at a time, I think pushing the merged state of the repo manifest would be less overwhelming upstream that PRing each change. E.g. we'd only approve this automated upstream PR for the last merged PR to a given docker repo to batch release changes. This is similar to what we do now manually.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand the question. Here is an example PR created by this script: ros-infrastructure/official-images#8 . Does the diff answer it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Your pr is a multiple merged distro changes, yet local PRs by the @osrf-docker-builder separate each distro PR, so the maintainer should be prudent in waiting for all the changes targeting a given repo manifest to be merged before upstreaming the final updated manifest. To avoid opening to many PR, they are batched into one periodic PR for that respective dockerhub repo.
Example PR:
#233
#234
#235
#236
#231
Upstream PR:
docker-library/official-images#5573