Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build less frequently #315

Closed
wants to merge 4 commits into from
Closed

Build less frequently #315

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
87defc8
add matrix-latest.json
colearendt May 19, 2022
679083a
gitignore python-version
colearendt May 19, 2022
32edf96
add a more robust computation of directory change
colearendt May 19, 2022
fded8a0
update ci to use the diff filter
colearendt May 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 21 additions & 5 deletions .github/workflows/build-content.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,24 +5,32 @@ on:
- dev
pull_request:

name: build/push RStudio Content Images
name: build/push (content)
jobs:

matrix:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v3
with:
python-version: '3.9'
architecture: 'x64'
- id: set-matrix
run: |
MATRIX=$(jq -Mcr < content/matrix.json)
echo "::set-output name=matrix::$MATRIX"
MATRIX_DIFF_FILTER=$(echo "$MATRIX" | ./get-diffs.py -i -t origin/main)
echo "Filtered Diff Matrix: $MATRIX_DIFF_FILTER"
echo "::set-output name=matrix::$MATRIX_DIFF_FILTER"

build:
runs-on: ubuntu-latest
needs: matrix
name: r${{ matrix.config.r }} py${{ matrix.config.py }} ${{ matrix.config.os }} ${{ github.ref }}
needs: matrix

strategy:
fail-fast: false
Expand All @@ -31,7 +39,15 @@ jobs:

steps:
- name: Check Out Repo
uses: actions/checkout@v2
uses: actions/checkout@v3

# setup python
- uses: actions/setup-python@v3
with:
python-version: '3.9'
architecture: 'x64'
cache: 'pip'
- run: pip install -r requirements.txt

- name: Set up Docker Buildx
id: buildx
Expand Down
44 changes: 34 additions & 10 deletions .github/workflows/build-latest.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,30 +3,54 @@ on:
branches:
- main
- dev
- dev-rspm
pull_request:

name: build/test/push (latest)
jobs:

matrix:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v3
with:
python-version: '3.9'
architecture: 'x64'
- id: set-matrix
run: |
# filter the matrix configuration
MATRIX=$(jq -Mcr < matrix-latest.json)
echo "Raw Matrix: $MATRIX"
MATRIX_DIFF_FILTER=$(echo "$MATRIX" | ./get-diffs.py -i -t origin/main)
echo "Filtered Diff Matrix: $MATRIX_DIFF_FILTER"
echo "::set-output name=matrix::$MATRIX_DIFF_FILTER"

build:
runs-on: ubuntu-latest
name: build-${{ matrix.config.tag_prefix}}${{ matrix.config.product}}
needs: matrix

strategy:
fail-fast: false
matrix:
config:
- {product: 'rstudio-workbench', dir: 'workbench', os: 'bionic/amd64', prefix: RSW, version: 'latest'}
- {product: 'rstudio-connect', dir: 'connect', prefix: RSC, version: 'latest'}
- {product: 'rstudio-connect-content-init', dir: 'connect-content-init', prefix: RSC, version: 'latest'}
- {product: 'rstudio-package-manager', dir: 'package-manager', prefix: RSPM, version: 'latest'}
- {product: 'r-session-complete', tag_prefix: 'bionic-', dir: 'r-session-complete/bionic', os: 'bionic/amd64', prefix: RSW, version: 'latest'}
- {product: 'r-session-complete', tag_prefix: 'centos7-', dir: 'r-session-complete/centos7', os: 'centos7/x86_64', prefix: RSW, version: 'latest'}
- {product: 'rstudio-workbench-for-microsoft-azure-ml', dir: 'helper/workbench-for-microsoft-azure-ml', os: 'bionic/amd64', prefix: RSW, version: 'latest'}
config: ${{ fromJson(needs.matrix.outputs.matrix) }}

steps:
- name: Check Out Repo
uses: actions/checkout@v2
uses: actions/checkout@v3

# setup python
- uses: actions/setup-python@v3
with:
python-version: '3.9'
architecture: 'x64'
cache: 'pip'
- run: pip install -r requirements.txt

- name: Set up Docker Buildx
id: buildx
Expand Down Expand Up @@ -117,7 +141,7 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.BUILD_PAT }}

- name: Push image(s) to Docker Hub
- name: Push image(s)
if: ${{ github.ref == 'refs/heads/main' }}
run: |
docker push rstudio/${{ matrix.config.product }}:${{ matrix.config.tag_prefix }}${{ matrix.config.version }}
Expand Down
18 changes: 16 additions & 2 deletions .github/workflows/build-preview-webhook.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,13 @@ jobs:
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v3
with:
python-version: '3.9'
architecture: 'x64'
- id: set-matrix
run: |
# filter the matrix configuration
Expand All @@ -43,7 +49,15 @@ jobs:

steps:
- name: Check Out Repo
uses: actions/checkout@v2
uses: actions/checkout@v3

# setup python
- uses: actions/setup-python@v3
with:
python-version: '3.9'
architecture: 'x64'
cache: 'pip'
- run: pip install -r requirements.txt

- name: Set up Docker Buildx
id: buildx
Expand Down
22 changes: 19 additions & 3 deletions .github/workflows/build-preview.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,13 @@ jobs:
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-python@v3
with:
python-version: '3.9'
architecture: 'x64'
- id: set-matrix
run: |
# filter the matrix configuration
Expand All @@ -28,7 +34,9 @@ jobs:
# NOTE: blank which input here means that every product is selected
MATRIX_FILTER=$(echo "$MATRIX" | jq -Mcr 'map(select((.product | startswith("${{ github.event.inputs.which }}"))or("all"=="${{ github.event.inputs.which }}")))')
echo "Filtered Matrix: $MATRIX_FILTER"
echo "::set-output name=matrix::$MATRIX_FILTER"
MATRIX_DIFF_FILTER=$(echo "$MATRIX_FILTER" | ./get-diffs.py -i -t origin/main)
echo "Filtered Diff Matrix: $MATRIX_DIFF_FILTER"
echo "::set-output name=matrix::$MATRIX_DIFF_FILTER"

build:
runs-on: ubuntu-latest
Expand All @@ -42,7 +50,15 @@ jobs:

steps:
- name: Check Out Repo
uses: actions/checkout@v2
uses: actions/checkout@v3

# setup python
- uses: actions/setup-python@v3
with:
python-version: '3.9'
architecture: 'x64'
cache: 'pip'
- run: pip install -r requirements.txt
Copy link
Contributor

@bschwedler bschwedler May 19, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we switch over to using urllib instead of requests so we can just use the standard library?

Copy link
Contributor Author

@colearendt colearendt May 19, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's definitely an option! Did I miss something? I thought requests was standard library at one time... but maybe that was old? Or I'm just ignorant and was using a "super standard library"

Copy link
Contributor

@bschwedler bschwedler May 19, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

python2 had urllib, urllib2, and urllib3. requests has an a nicer interface with which to interact, but since we are using it minimally it should be easy to replace with urllib in py3.


- name: Set up Docker Buildx
id: buildx
Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ data
*.lic
.idea/
tmp*/
.python-version
106 changes: 94 additions & 12 deletions get-diffs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
# --all, -a Whether to bypass checks and return the input
# --dirs DIRS [DIRS ...], -d DIRS [DIRS ...]
# A subset of directories to check for differences

import pathlib
import re
import json
import argparse
Expand All @@ -41,15 +41,15 @@ def uniq_list(input: typing.List[typing.Any]) -> typing.List:
return output


def get_changed_dirs(commit : str, dirs=None) -> typing.List[str]:
def get_changed_dirs(commit: str, dirs: typing.List[str] = None) -> typing.List[str]:
if dirs is None:
dirs = []
changed_files = get_changed_files(commit, dirs)
changed_dirs = [os.path.dirname(f) for f in changed_files]
return uniq_list(changed_dirs)


def get_changed_files(commit : str, dirs=None) -> typing.List[str]:
def get_changed_files(commit: str, dirs: typing.List[str] = None) -> typing.List[str]:
if dirs is None:
dirs = []
command = ['git', 'diff', '--name-only', '--find-renames', commit, '--'] + dirs
Expand All @@ -64,17 +64,43 @@ def get_current_commit() -> str:
return res.decode('utf-8').strip().replace("'", '')


def get_merge_base(commit1 : str, commit2 : str) -> str:
def get_parent_commit(n: int = 1) -> str:
res = subprocess.check_output(['git', 'log', '-n', 1, '--pretty=%P', '|', 'cut', '-f', n])
return res.decode('utf-8').strip().replace("'", '')


def get_merge_base(commit1: str, commit2: str) -> str:
print(commit2, file=sys.stderr)
res = subprocess.check_output(['git', 'merge-base', commit1, commit2])
return res.decode('utf-8').strip()


def filter_json_by_dirs(json_input: typing.List[typing.Dict], dirs=None) -> typing.List[dict]:
def is_dir_changed(input_dir: str, changed_dirs: typing.List[str] = None, include_parent: bool = False) -> bool:
# NOTE: python3.9+ feature only
i = pathlib.PurePath(input_dir)
if changed_dirs is None:
return False
else:
for changed_dir in changed_dirs:
# NOTE: python3.9+ feature only
p = pathlib.PurePath(changed_dir)

# if a subdirectory changed
if p.is_relative_to(input_dir):
return True

# also if a parent directory changed
if include_parent & i.is_relative_to(changed_dir):
return True
return False


def filter_json_by_dirs(json_input: typing.List[typing.Dict], dirs: typing.List[str] = None) -> typing.List[dict]:
if dirs is None:
dirs = []
output_data = []
for m in json_input:
if m['dir'] in dirs:
if is_dir_changed(m['dir'], dirs):
output_data.append(m)
return output_data

Expand All @@ -84,6 +110,21 @@ def get_dirs_from_json(json_input: typing.List[typing.Dict]) -> typing.List[str]
return uniq_list(base_data)


def any_important_changed_files(commit: str, important_files: typing.List[str]) -> bool:
all_changed_files = get_changed_files(mb, ['.'])
important_changed_files = []
print(f"Changed files: {all_changed_files}", file=sys.stderr)
for i_file in important_files:
# any important_file sub-path of changed
# NOTE: python3.9+ feature only
if any([pathlib.PurePath(f).is_relative_to(i_file) for f in all_changed_files]):
important_changed_files.append(i_file)
if len(important_changed_files) > 0:
print(f"Important changed files. Returning all diffs: {important_changed_files}", file=sys.stderr)
return True
return False


if __name__ == "__main__":

# ------------------------------------------
Expand All @@ -103,9 +144,28 @@ def get_dirs_from_json(json_input: typing.List[typing.Dict]) -> typing.List[str]
)
parser.add_argument(
"--all", "-a",
action="store_true",
type=str,
nargs='?',
default=['false'],
const=['true'],
help="Whether to bypass checks and return the input",
)
parser.add_argument(
"--target", "-t",
type=str,
nargs=1,
default=['main'],
help="The merge target to reference",
)
# uses type "str" to look for "true" (and play nicely with GHA)
parser.add_argument(
"--use-parent", "-p",
type=str,
nargs='?',
default=['false'],
const=['true'],
help="Whether to use the first parent commit as the target (when 'true')",
)
parser.add_argument(
"--dirs", "-d",
type=str,
Expand All @@ -118,7 +178,9 @@ def get_dirs_from_json(json_input: typing.List[typing.Dict]) -> typing.List[str]

file = args.file
dirs = args.dirs
return_all = args.all
target = args.target[0]
return_all = len(args.all) == 0 or args.all[0].lower() == 'true'
use_parent = len(args.use_parent) == 0 or args.use_parent[0].lower() == 'true'
read_stdin = args.stdin

# ----------------------------------------------------------
Expand Down Expand Up @@ -159,15 +221,22 @@ def get_dirs_from_json(json_input: typing.List[typing.Dict]) -> typing.List[str]

if return_all:
print('Returning input plus filters, not computing diff, due to -a/--all', file=sys.stderr)
print(filter_json_by_dirs(matrix_data, directories_base))
if len(directories_base) == 0:
print(matrix_data)
else:
print(filter_json_by_dirs(matrix_data, directories_base))
exit(0)

# ----------------------------------------------------------
# Determine which base directories have diffs
# ----------------------------------------------------------

if use_parent:
target = get_parent_commit(1)
print(f"Using parent commit to overwrite the `target` (-p / --parent): {target}")

cc = get_current_commit()
mb = get_merge_base(cc, 'main')
mb = get_merge_base(cc, target)
print(f'Current commit: {cc}', file=sys.stderr)
print(f"Merge Base: {mb}", file=sys.stderr)
changed_directories = get_changed_dirs(mb, directories_base)
Expand All @@ -176,6 +245,19 @@ def get_dirs_from_json(json_input: typing.List[typing.Dict]) -> typing.List[str]
changed_directories_no_root = [d for d in changed_directories if len(d) > 0]
print(f"Changed directories: {changed_directories_no_root}", file=sys.stderr)

output_structure = []
print(filter_json_by_dirs(matrix_data, changed_directories_no_root))
# ----------------------------------------------------------
# Determine if any important diffs in the root directory (ci, etc.)
# ----------------------------------------------------------

# these are "shared resources" that get used in the build pipeline
important_files = [
'.github/workflows',
'get-diffs.py', 'get-version.py',
'matrix-preview.json', 'matrix-latest.json',
'content/matrix.json'
]
if any_important_changed_files(mb, important_files):
print(matrix_data)
else:
print(filter_json_by_dirs(matrix_data, changed_directories))
exit(0)
Loading