Skip to content

Commit

Permalink
ci(build-depends.repos): separate repos for humble and nightly (autow…
Browse files Browse the repository at this point in the history
…arefoundation#10148)

Signed-off-by: Mete Fatih Cırıt <[email protected]>
  • Loading branch information
xmfcx authored Feb 18, 2025
1 parent aa19eb9 commit 0674e20
Show file tree
Hide file tree
Showing 12 changed files with 241 additions and 35 deletions.
43 changes: 43 additions & 0 deletions .github/actions/combine-repos-action/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# combine-repos-action

## Description

**Combine Repos Action** is a GitHub Action designed to merge two `.repos` files—a base file and an overlay file—into a single file. The overlay file takes precedence over the base file when duplicate repository entries exist. This is especially useful for projects that need to dynamically combine configuration files for dependency management or CI workflows.

## Usage

Below is an example of how to include it in your workflow:

```yaml
jobs:
combine:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Combine Repos Files
uses: ./.github/actions/combine-repos-action
with:
base_file: build_depends_humble.repos
overlay_file: build_depends_nightly.repos
output_file: build_depends.repos
```
In this example:
- The action reads the `build_depends_humble.repos` file and the `build_depends_nightly.repos` file.
- It merges them with overlay file taking precedence.
- The resulting file is saved as `build_depends.repos` (or a custom filename if specified).

## Inputs

| Input | Description | Required | Default |
| -------------- | --------------------------------- | -------- | ---------------- |
| `base_file` | Path to the base `.repos` file | Yes | - |
| `overlay_file` | Path to the overlay `.repos` file | Yes | - |
| `output_file` | Path for the combined output file | No | `combined.repos` |

## Outputs

This action creates or updates the file specified by the `output_file` input with the combined contents of the two `.repos` files. The final file's content is also echoed to the workflow logs for easy verification.
48 changes: 48 additions & 0 deletions .github/actions/combine-repos-action/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
name: Combine Repos Action
description: Merge base and overlay .repos files, with overlay entries taking precedence.
inputs:
base_file:
description: Path to the base .repos file
required: true
overlay_file:
description: Path to the overlay .repos file
required: true
output_file:
description: Path for the combined output file
required: false
default: combined.repos

runs:
using: composite
steps:
- name: Install python3-pip
run: |
sudo apt-get -yqq update
sudo apt-get -yqq install python3-pip python-is-python3
shell: bash

- name: Display Python version
run: |
python --version
which pip
pip --version
shell: bash

- name: Install PyYAML dependency
run: pip install pyyaml
shell: bash

- name: Combine repos files
run: |
python "${GITHUB_ACTION_PATH}/combine-repos.py" \
--base "${{ inputs.base_file }}" \
--overlay "${{ inputs.overlay_file }}" \
--output "${{ inputs.output_file }}"
shell: bash

- name: Display combined repos file
run: |
echo "=== Combined .repos File Content ==="
cat "${{ inputs.output_file }}"
echo "===================================="
shell: bash
52 changes: 52 additions & 0 deletions .github/actions/combine-repos-action/combine-repos.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
#!/usr/bin/env python3
import argparse
import sys

import yaml


def main():
parser = argparse.ArgumentParser(
description="Merge base and overlay .repos files with overlay taking precedence."
)
parser.add_argument("--base", required=True, help="Path to the base .repos file")
parser.add_argument("--overlay", required=True, help="Path to the overlay .repos file")
parser.add_argument(
"--output",
default="combined.repos",
help="Path for the combined output file (default: combined.repos)",
)
args = parser.parse_args()

try:
with open(args.base, "r") as bf:
base_data = yaml.safe_load(bf)
except Exception as e:
sys.exit(f"Error reading base file '{args.base}': {e}")

try:
with open(args.overlay, "r") as of:
overlay_data = yaml.safe_load(of)
except Exception as e:
sys.exit(f"Error reading overlay file '{args.overlay}': {e}")

if "repositories" not in base_data:
sys.exit(f"Base file '{args.base}' is missing the 'repositories' key")
if overlay_data and "repositories" not in overlay_data:
sys.exit(f"Overlay file '{args.overlay}' is missing the 'repositories' key")

# Merge: overlay entries override base entries
merged_data = base_data.copy()
merged_data["repositories"].update(overlay_data.get("repositories", {}))

try:
with open(args.output, "w") as cf:
yaml.dump(merged_data, cf, default_flow_style=False)
except Exception as e:
sys.exit(f"Error writing to output file '{args.output}': {e}")

print(f"Successfully merged into {args.output}")


if __name__ == "__main__":
main()
1 change: 0 additions & 1 deletion .github/sync-files.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
dest: .github/pull_request_template.md
- source: .github/stale.yml
- source: .github/workflows/cancel-previous-workflows.yaml
- source: .github/workflows/check-build-depends.yaml
- source: .github/workflows/clang-tidy-pr-comments.yaml
- source: .github/workflows/clang-tidy-pr-comments-manually.yaml
- source: .github/workflows/comment-on-pr.yaml
Expand Down
12 changes: 9 additions & 3 deletions .github/workflows/build-and-test-daily-arm64.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ jobs:
include:
- rosdistro: humble
container: ghcr.io/autowarefoundation/autoware:universe-devel
build-depends-repos: build_depends.repos
steps:
- name: Check out repository
uses: actions/checkout@v4
Expand Down Expand Up @@ -54,13 +53,20 @@ jobs:
echo "::notice::BUILD_TYPE_CUDA_STATE=$build_type_cuda_state"
shell: bash

- name: Prepare build_depends.repos file
uses: ./.github/actions/combine-repos-action
with:
base_file: build_depends_humble.repos
overlay_file: build_depends_nightly.repos
output_file: build_depends.repos

- name: Build
if: ${{ steps.get-self-packages.outputs.self-packages != '' }}
uses: autowarefoundation/autoware-github-actions/colcon-build@v1
with:
rosdistro: ${{ matrix.rosdistro }}
target-packages: ${{ steps.get-self-packages.outputs.self-packages }}
build-depends-repos: ${{ matrix.build-depends-repos }}
build-depends-repos: build_depends.repos
cache-key-element: ${{ env.BUILD_TYPE_CUDA_STATE }}
build-pre-command: taskset --cpu-list 0-6

Expand All @@ -71,7 +77,7 @@ jobs:
with:
rosdistro: ${{ matrix.rosdistro }}
target-packages: ${{ steps.get-self-packages.outputs.self-packages }}
build-depends-repos: ${{ matrix.build-depends-repos }}
build-depends-repos: build_depends.repos

- name: Upload coverage to CodeCov
if: ${{ steps.test.outputs.coverage-report-files != '' }}
Expand Down
12 changes: 9 additions & 3 deletions .github/workflows/build-and-test-daily.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ jobs:
include:
- rosdistro: humble
container: ghcr.io/autowarefoundation/autoware:universe-devel
build-depends-repos: build_depends.repos
steps:
- name: Check out repository
uses: actions/checkout@v4
Expand Down Expand Up @@ -81,13 +80,20 @@ jobs:
echo "::notice::BUILD_TYPE_CUDA_STATE=$build_type_cuda_state"
shell: bash

- name: Prepare build_depends.repos file
uses: ./.github/actions/combine-repos-action
with:
base_file: build_depends_humble.repos
overlay_file: build_depends_nightly.repos
output_file: build_depends.repos

- name: Build
if: ${{ steps.get-self-packages.outputs.self-packages != '' }}
uses: autowarefoundation/autoware-github-actions/colcon-build@v1
with:
rosdistro: ${{ matrix.rosdistro }}
target-packages: ${{ steps.get-self-packages.outputs.self-packages }}
build-depends-repos: ${{ matrix.build-depends-repos }}
build-depends-repos: build_depends.repos
cache-key-element: ${{ env.BUILD_TYPE_CUDA_STATE }}

- name: Show ccache stats after build
Expand All @@ -101,7 +107,7 @@ jobs:
with:
rosdistro: ${{ matrix.rosdistro }}
target-packages: ${{ steps.get-self-packages.outputs.self-packages }}
build-depends-repos: ${{ matrix.build-depends-repos }}
build-depends-repos: build_depends.repos

- name: Upload coverage to CodeCov
if: ${{ steps.test.outputs.coverage-report-files != '' }}
Expand Down
18 changes: 15 additions & 3 deletions .github/workflows/build-and-test-differential-arm64.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@ jobs:
include:
- rosdistro: humble
container: ghcr.io/autowarefoundation/autoware:universe-devel
build-depends-repos: build_depends.repos
steps:
- name: Set PR fetch depth
run: echo "PR_FETCH_DEPTH=$(( ${{ github.event.pull_request.commits }} + 1 ))" >> "${GITHUB_ENV}"
Expand Down Expand Up @@ -68,13 +67,26 @@ jobs:
echo "::notice::BUILD_TYPE_CUDA_STATE=$build_type_cuda_state"
shell: bash

- name: Prepare build_depends.repos file (main branch)
if: ${{ github.event.pull_request.base.ref == 'main' }}
uses: ./.github/actions/combine-repos-action
with:
base_file: build_depends_humble.repos
overlay_file: build_depends_nightly.repos
output_file: build_depends.repos

- name: Prepare build_depends.repos file (humble branch)
if: ${{ github.event.pull_request.base.ref == 'humble' }}
run: cp build_depends_humble.repos build_depends.repos
shell: bash

- name: Build
if: ${{ steps.get-modified-packages.outputs.modified-packages != '' }}
uses: autowarefoundation/autoware-github-actions/colcon-build@v1
with:
rosdistro: ${{ matrix.rosdistro }}
target-packages: ${{ steps.get-modified-packages.outputs.modified-packages }}
build-depends-repos: ${{ matrix.build-depends-repos }}
build-depends-repos: build_depends.repos
cache-key-element: ${{ env.BUILD_TYPE_CUDA_STATE }}
build-pre-command: taskset --cpu-list 0-5

Expand All @@ -85,7 +97,7 @@ jobs:
with:
rosdistro: ${{ matrix.rosdistro }}
target-packages: ${{ steps.get-modified-packages.outputs.modified-packages }}
build-depends-repos: ${{ matrix.build-depends-repos }}
build-depends-repos: build_depends.repos

- name: Upload coverage to CodeCov
if: ${{ steps.test.outputs.coverage-report-files != '' }}
Expand Down
13 changes: 13 additions & 0 deletions .github/workflows/build-and-test-differential.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,19 @@ jobs:
echo "::notice::BUILD_TYPE_CUDA_STATE=$build_type_cuda_state"
shell: bash

- name: Prepare build_depends.repos file (main branch)
if: ${{ github.event.pull_request.base.ref == 'main' }}
uses: ./.github/actions/combine-repos-action
with:
base_file: build_depends_humble.repos
overlay_file: build_depends_nightly.repos
output_file: build_depends.repos

- name: Prepare build_depends.repos file (humble branch)
if: ${{ github.event.pull_request.base.ref == 'humble' }}
run: cp build_depends_humble.repos build_depends.repos
shell: bash

- name: Build
if: ${{ steps.get-modified-packages.outputs.modified-packages != '' }}
uses: autowarefoundation/autoware-github-actions/colcon-build@v1
Expand Down
12 changes: 9 additions & 3 deletions .github/workflows/build-and-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ jobs:
include:
- rosdistro: humble
container: ghcr.io/autowarefoundation/autoware:universe-devel
build-depends-repos: build_depends.repos
steps:
- name: Check out repository
uses: actions/checkout@v4
Expand Down Expand Up @@ -85,13 +84,20 @@ jobs:
echo "::notice::BUILD_TYPE_CUDA_STATE=$build_type_cuda_state"
shell: bash

- name: Prepare build_depends.repos file
uses: ./.github/actions/combine-repos-action
with:
base_file: build_depends_humble.repos
overlay_file: build_depends_nightly.repos
output_file: build_depends.repos

- name: Build
if: ${{ steps.get-self-packages.outputs.self-packages != '' }}
uses: autowarefoundation/autoware-github-actions/colcon-build@v1
with:
rosdistro: ${{ matrix.rosdistro }}
target-packages: ${{ steps.get-self-packages.outputs.self-packages }}
build-depends-repos: ${{ matrix.build-depends-repos }}
build-depends-repos: build_depends.repos
cache-key-element: ${{ env.BUILD_TYPE_CUDA_STATE }}
build-pre-command: taskset --cpu-list 0-5

Expand All @@ -115,7 +121,7 @@ jobs:
with:
rosdistro: ${{ matrix.rosdistro }}
target-packages: ${{ steps.get-self-packages.outputs.self-packages }}
build-depends-repos: ${{ matrix.build-depends-repos }}
build-depends-repos: build_depends.repos

- name: Upload coverage to CodeCov
if: ${{ steps.test.outputs.coverage-report-files != '' }}
Expand Down
20 changes: 14 additions & 6 deletions .github/workflows/check-build-depends.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
# This file is automatically synced from:
# https://github.com/autowarefoundation/sync-file-templates
# To make changes, update the source repository and follow the guidelines in its README.

name: check-build-depends

on:
Expand All @@ -21,7 +17,6 @@ jobs:
include:
- rosdistro: humble
container: ros:humble
build-depends-repos: build_depends.repos
steps:
- name: Check out repository
uses: actions/checkout@v4
Expand All @@ -33,9 +28,22 @@ jobs:
id: get-self-packages
uses: autowarefoundation/autoware-github-actions/get-self-packages@v1

- name: Prepare build_depends.repos file (main branch)
if: ${{ github.event.pull_request.base.ref == 'main' }}
uses: ./.github/actions/combine-repos-action
with:
base_file: build_depends_humble.repos
overlay_file: build_depends_nightly.repos
output_file: build_depends.repos

- name: Prepare build_depends.repos file (humble branch)
if: ${{ github.event.pull_request.base.ref == 'humble' }}
run: cp build_depends_humble.repos build_depends.repos
shell: bash

- name: Build
uses: autowarefoundation/autoware-github-actions/colcon-build@v1
with:
rosdistro: ${{ matrix.rosdistro }}
target-packages: ${{ steps.get-self-packages.outputs.self-packages }}
build-depends-repos: ${{ matrix.build-depends-repos }}
build-depends-repos: build_depends.repos
Loading

0 comments on commit 0674e20

Please sign in to comment.