-
Notifications
You must be signed in to change notification settings - Fork 546
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Toolkit: provide mechanism to auto-install build deps when building on azl #11081
Open
dmcilvaney
wants to merge
6
commits into
microsoft:3.0-dev
Choose a base branch
from
dmcilvaney:damcilva/3.0/tools/auto-prereq
base: 3.0-dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+312
−2
Open
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
aed5cd3
Automate prereq install
dmcilvaney 7383d15
drop systemd for independent PRs
dmcilvaney 4f880a0
Add comments to json, clean up script
dmcilvaney cc1716d
Clarify utility, support "mariner", auto-detect from /etc/os-release
dmcilvaney 5c18a29
drop old branches
dmcilvaney 4127cf0
fix whitepsace
dmcilvaney File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # Copyright (c) Microsoft Corporation. | ||
| # Licensed under the MIT License. | ||
|
|
||
| # This action checks that the licenses.json file is up-to-date | ||
| # and that the LICENSES-MAP.md file is up-to-date | ||
| name: Prerequisites Check | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main, 2.0*, 3.0*, fasttrack/*] | ||
| pull_request: | ||
| branches: [main, 2.0*, 3.0*, fasttrack/*] | ||
|
|
||
| jobs: | ||
| check: | ||
| name: Prerequisites Check | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| # Checkout the branch of our repo that triggered this action | ||
| - name: Workflow trigger checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Run prerequisites checking script for Ubuntu | ||
| run: | | ||
| if [ ! -f ./toolkit/docs/building/prerequisites-src.json ]; then | ||
| echo "prerequisites-src.json not found" | ||
| exit 0 | ||
| fi | ||
| ./toolkit/scripts/prerequisites.sh \ | ||
| -m ./toolkit/docs/building/prerequisites-ubuntu.md \ | ||
| -s ./toolkit/docs/building/prerequisites-src.json \ | ||
| -d "ubuntu" | ||
|
|
||
| - name: Run prerequisites checking script for AzureLinux | ||
| run: | | ||
| if [ ! -f ./toolkit/docs/building/prerequisites-src.json ]; then | ||
| echo "prerequisites-src.json not found" | ||
| exit 0 | ||
| fi | ||
| ./toolkit/scripts/prerequisites.sh \ | ||
| -m ./toolkit/docs/building/prerequisites-mariner.md \ | ||
| -s ./toolkit/docs/building/prerequisites-src.json \ | ||
| -d "azurelinux" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| { | ||
| "_comment": "This JSON file lists the prerequisites for building the source code in the CBL-Mariner toolkit.", | ||
| "_comment2": "check-prerequisites.yml and 'make install-azurelinux-prereqs' use this file as the canonical truth", | ||
| "_comment3": "The 'name' field is the name of the package to install, followed by the actual package name for each supported OS", | ||
| "packages": [ | ||
| { | ||
| "name": "acl", | ||
| "azurelinux": "acl", | ||
| "ubuntu": "acl" | ||
| }, | ||
| { | ||
| "name": "binutils", | ||
| "azurelinux": "binutils" | ||
| }, | ||
| { | ||
| "name": "cdrkit", | ||
| "azurelinux": "cdrkit" | ||
| }, | ||
| { | ||
| "name": "curl", | ||
| "azurelinux": "curl", | ||
| "ubuntu": "curl" | ||
| }, | ||
| { | ||
| "name": "diffutils", | ||
| "azurelinux": "diffutils", | ||
| "ubuntu": "diffutils" | ||
| }, | ||
| { | ||
| "name": "dosfstools", | ||
| "azurelinux": "dosfstools" | ||
| }, | ||
| { | ||
| "name": "gawk", | ||
| "azurelinux": "gawk", | ||
| "ubuntu": "gawk" | ||
| }, | ||
| { | ||
| "name": "glibc-devel", | ||
| "azurelinux": "glibc-devel" | ||
| }, | ||
| { | ||
| "name": "genisoimage", | ||
| "azurelinux": "genisoimage", | ||
| "ubuntu": "genisoimage" | ||
| }, | ||
| { | ||
| "name": "git", | ||
| "azurelinux": "git", | ||
| "ubuntu": "git" | ||
| }, | ||
| { | ||
| "name": "golang", | ||
| "azurelinux": "golang", | ||
| "ubuntu": "golang-1.21-go" | ||
| }, | ||
| { | ||
| "name": "jq", | ||
| "azurelinux": "jq", | ||
| "ubuntu": "jq" | ||
| }, | ||
| { | ||
| "name": "kernel-headers", | ||
| "azurelinux": "kernel-headers" | ||
| }, | ||
| { | ||
| "name": "make", | ||
| "azurelinux": "make", | ||
| "ubuntu": "make" | ||
| }, | ||
| { | ||
| "name": "moby-cli", | ||
| "azurelinux": "moby-cli" | ||
| }, | ||
| { | ||
| "name": "moby-engine", | ||
| "azurelinux": "moby-engine" | ||
| }, | ||
| { | ||
| "name": "openssl", | ||
| "azurelinux": "openssl", | ||
| "ubuntu": "openssl" | ||
| }, | ||
| { | ||
| "name": "parted", | ||
| "azurelinux": "parted", | ||
| "ubuntu": "parted" | ||
| }, | ||
| { | ||
| "name": "pigz", | ||
| "azurelinux": "pigz", | ||
| "ubuntu": "pigz" | ||
| }, | ||
| { | ||
| "name": "qemu-img", | ||
| "azurelinux": "qemu-img", | ||
| "ubuntu": "qemu-utils" | ||
| }, | ||
| { | ||
| "name": "rpm", | ||
| "azurelinux": "rpm", | ||
| "ubuntu": "rpm" | ||
| }, | ||
| { | ||
| "name": "rpm-build", | ||
| "azurelinux": "rpm-build" | ||
| }, | ||
| { | ||
| "name": "sudo", | ||
| "azurelinux": "sudo" | ||
| }, | ||
| { | ||
| "name": "tar", | ||
| "azurelinux": "tar", | ||
| "ubuntu": "tar" | ||
| }, | ||
| { | ||
| "name": "wget", | ||
| "azurelinux": "wget", | ||
| "ubuntu": "wget" | ||
| }, | ||
| { | ||
| "name": "xfsprogs", | ||
| "azurelinux": "xfsprogs", | ||
| "ubuntu": "xfsprogs" | ||
| }, | ||
| { | ||
| "name": "zstd", | ||
| "azurelinux": "zstd", | ||
| "ubuntu": "zstd" | ||
| } | ||
| ] | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Copyright (c) Microsoft Corporation. | ||
| # Licensed under the MIT License. | ||
|
|
||
| set -e | ||
|
|
||
| prereq_md_path="" | ||
| prereq_json_path="" | ||
|
|
||
| function print_help { | ||
| echo "Build prereq utility. Lists or validates pre-requisites for a given distro." | ||
| echo "Usage:" | ||
| echo '[MANDATORY] -s PATH -> path to the src json file' | ||
| echo '[MANDATORY] -d DISTRO -> One of "mariner", "azurelinux" or "ubuntu"' | ||
| echo '[OPTIONAL] -m PATH -> path to the pre-requisite md file, must be provided if not in print mode' | ||
| echo '[OPTIONAL] -p -> print the canonical pre-requisite list and exit' | ||
| echo '[OPTIONAL] -h -> print this help dialogue and exit' | ||
| } | ||
|
|
||
| do_print=false | ||
| while getopts "hpm:s:d:" OPTIONS; do | ||
| case ${OPTIONS} in | ||
| m ) prereq_md_path=$OPTARG ;; | ||
| s ) prereq_json_path=$OPTARG ;; | ||
| d ) distro=$OPTARG ;; | ||
| p ) do_print="true" ;; | ||
| h ) print_help; exit 0 ;; | ||
| ? ) echo -e "ERROR: INVALID OPTION.\n\n" >&2; print_help; exit 1 ;; | ||
| esac | ||
| done | ||
|
|
||
| if [ -z "$prereq_json_path" ]; then | ||
| echo "ERROR: Missing src json file path." >&2 | ||
| print_help | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [ ! -f "$prereq_json_path" ]; then | ||
| echo "ERROR: Src json file not found at $prereq_json_path." >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [ -z "$distro" ]; then | ||
| echo "ERROR: Missing distro." >&2 | ||
| print_help | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [ "$distro" != "mariner" ] && [ "$distro" != "azurelinux" ] && [ "$distro" != "ubuntu" ]; then | ||
| echo "ERROR: Invalid distro. Must be one of 'azurelinux' or 'ubuntu'." >&2 | ||
| print_help | ||
| exit 1 | ||
| fi | ||
| if [ "$distro" == "mariner" ]; then | ||
| distro="azurelinux" | ||
| fi | ||
|
|
||
| # Load the pre-requisite source json for the distro of interest | ||
| # Json is of the form: | ||
| # { | ||
| # "packages": [ | ||
| # { | ||
| # "name": "pkg", | ||
| # "azurelinux": "pkg-b", | ||
| # "ubuntu": "pkg-a" | ||
| # }, | ||
| # ... | ||
| # ] | ||
| #} | ||
| # A distro specific entry may not exist for every package, so get the entries that do exist, and | ||
| # select the distro specific value. Convert the result to a space separated string. | ||
| src_prereqs=$(jq -r --arg distro "$distro" '[.packages[] | select(has($distro)) | .[$distro]] | sort | join(" ")' "$prereq_json_path") | ||
|
|
||
| # For print mode, just print the pre-requisites and exit early. | ||
| if "$do_print"; then | ||
| echo "$src_prereqs" | ||
| exit 0 | ||
| fi | ||
|
|
||
| if [ -z "$prereq_md_path" ]; then | ||
| echo "ERROR: Missing pre-requisite file path." >&2 | ||
| print_help | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [ ! -f "$prereq_md_path" ]; then | ||
| echo "ERROR: Pre-requisite file not found at $prereq_md_path." >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Get the current entry in the .md file: | ||
| # 1) Remove all line continuations (backslash followed by newline) and replace with a space | ||
| current_prereqs="$(sed -e ':a' -e 'N' -e '$!ba' -e 's/\\\n/ /g' "$prereq_md_path")" | ||
|
|
||
| # 2) Remove all leading and trailing whitespace | ||
| current_prereqs=$(echo "$current_prereqs" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//') | ||
|
|
||
| # 3) Find the line that starts with "sudo <pkg manager> -y install ..." | ||
| if [[ "$distro" == "azurelinux" ]]; then | ||
| current_prereqs=$(echo "$current_prereqs" | grep -oP 'sudo tdnf -y install \K.*') | ||
| elif [[ "$distro" == "ubuntu" ]]; then | ||
| current_prereqs=$(echo "$current_prereqs" | grep -oP 'sudo apt -y install \K.*') | ||
| fi | ||
|
|
||
| # 4) Grab the package list from the line | ||
| current_prereqs=$(echo "$current_prereqs" | xargs) | ||
|
|
||
| # Compare the current and source pre-requisites. Enforce a 1:1 match including order for consistency. | ||
| if [ "$current_prereqs" != "$src_prereqs" ]; then | ||
| echo "ERROR: Pre-requisites do not match. Please update the pre-requisite file." | ||
| echo "" | ||
| echo "Comparing: $prereq_json_path" | ||
| echo "with: $prereq_md_path" | ||
| echo "" | ||
| echo ".json file: $src_prereqs" | ||
| echo ".md file: $current_prereqs" | ||
| exit 1 | ||
| else | ||
| echo "Pre-requisites match." | ||
| exit 0 | ||
| fi | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This script seems overly complicated.
Would it make sense to have
install-prerequisites-ubuntu.shandinstall-prerequisites-mariner.shscripts and then just have the Markdown reference those 2 scripts? That way, we still have a single source of truth but with significantly fewer lines of code.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The "real" install has some extra stuff I really don't want to deal with... this is for use with pipelines first and foremost. I'd be open to just removing the makefile target completly and having the pipeline call the script directly.