Skip to content

Commit 500774b

Browse files
maxulyssemaxulysse
authored
Merge branch 'dev' into fix_references_usage
2 parents c0dbaa5 + 370ba86 commit 500774b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

70 files changed

+2396
-1126
lines changed

‎.github/CONTRIBUTING.md

+6-6
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ If you'd like to write some code for nf-core/rnavar, the standard workflow is as
1919
1. Check that there isn't already an issue about your idea in the [nf-core/rnavar issues](https://github.com/nf-core/rnavar/issues) to avoid duplicating work. If there isn't one already, please create one so that others know you're working on this
2020
2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [nf-core/rnavar repository](https://github.com/nf-core/rnavar) to your GitHub account
2121
3. Make the necessary changes / additions within your forked repository following [Pipeline conventions](#pipeline-contribution-conventions)
22-
4. Use `nf-core schema build` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10).
22+
4. Use `nf-core pipelines schema build` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10).
2323
5. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged
2424

2525
If you're not used to this workflow with git, you can start with some [docs from GitHub](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests) or even their [excellent `git` resources](https://try.github.io/).
@@ -40,7 +40,7 @@ There are typically two types of tests that run:
4040
### Lint tests
4141

4242
`nf-core` has a [set of guidelines](https://nf-co.re/developers/guidelines) which all pipelines must adhere to.
43-
To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint <pipeline-directory>` command.
43+
To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core pipelines lint <pipeline-directory>` command.
4444

4545
If any failures or warnings are encountered, please follow the listed URL for more documentation.
4646

@@ -75,7 +75,7 @@ If you wish to contribute a new step, please use the following coding standards:
7575
2. Write the process block (see below).
7676
3. Define the output channel if needed (see below).
7777
4. Add any new parameters to `nextflow.config` with a default (see below).
78-
5. Add any new parameters to `nextflow_schema.json` with help text (via the `nf-core schema build` tool).
78+
5. Add any new parameters to `nextflow_schema.json` with help text (via the `nf-core pipelines schema build` tool).
7979
6. Add sanity checks and validation for all relevant parameters.
8080
7. Perform local tests to validate that the new code works as expected.
8181
8. If applicable, add a new test command in `.github/workflow/ci.yml`.
@@ -86,11 +86,11 @@ If you wish to contribute a new step, please use the following coding standards:
8686

8787
Parameters should be initialised / defined with default values in `nextflow.config` under the `params` scope.
8888

89-
Once there, use `nf-core schema build` to add to `nextflow_schema.json`.
89+
Once there, use `nf-core pipelines schema build` to add to `nextflow_schema.json`.
9090

9191
### Default processes resource requirements
9292

93-
Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/master/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.
93+
Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/main/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.
9494

9595
The process resources can be passed on to the tool dynamically within the process with the `${task.cpus}` and `${task.memory}` variables in the `script:` block.
9696

@@ -103,7 +103,7 @@ Please use the following naming schemes, to make it easy to understand what is g
103103

104104
### Nextflow version bumping
105105

106-
If you are using a new feature from core Nextflow, you may bump the minimum required version of nextflow in the pipeline with: `nf-core bump-version --nextflow . [min-nf-version]`
106+
If you are using a new feature from core Nextflow, you may bump the minimum required version of nextflow in the pipeline with: `nf-core pipelines bump-version --nextflow . [min-nf-version]`
107107

108108
### Images and figures
109109

‎.github/PULL_REQUEST_TEMPLATE.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-core/rnav
1717
- [ ] If you've fixed a bug or added code that should be tested, add tests!
1818
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-core/rnavar/tree/master/.github/CONTRIBUTING.md)
1919
- [ ] If necessary, also make a PR on the nf-core/rnavar _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
20-
- [ ] Make sure your code lints (`nf-core lint`).
20+
- [ ] Make sure your code lints (`nf-core pipelines lint`).
2121
- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
2222
- [ ] Check for unexpected warnings in debug mode (`nextflow run . -profile debug,test,docker --outdir <OUTDIR>`).
2323
- [ ] Usage Documentation in `docs/usage.md` is updated.

‎.github/workflows/awsfulltest.yml

+21-4
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,35 @@
11
name: nf-core AWS full size tests
2-
# This workflow is triggered on published releases.
2+
# This workflow is triggered on PRs opened against the master branch.
33
# It can be additionally triggered manually with GitHub actions workflow dispatch button.
44
# It runs the -profile 'test_full' on AWS batch
55

66
on:
7-
release:
8-
types: [published]
7+
pull_request:
8+
branches:
9+
- master
910
workflow_dispatch:
11+
pull_request_review:
12+
types: [submitted]
13+
1014
jobs:
1115
run-platform:
1216
name: Run AWS full tests
13-
if: github.repository == 'nf-core/rnavar'
17+
# run only if the PR is approved by at least 2 reviewers and against the master branch or manually triggered
18+
if: github.repository == 'nf-core/rnavar' && github.event.review.state == 'approved' && github.event.pull_request.base.ref == 'master' || github.event_name == 'workflow_dispatch'
1419
runs-on: ubuntu-latest
1520
steps:
21+
- uses: octokit/[email protected]
22+
id: check_approvals
23+
with:
24+
route: GET /repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/reviews
25+
env:
26+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
27+
- id: test_variables
28+
if: github.event_name != 'workflow_dispatch'
29+
run: |
30+
JSON_RESPONSE='${{ steps.check_approvals.outputs.data }}'
31+
CURRENT_APPROVALS_COUNT=$(echo $JSON_RESPONSE | jq -c '[.[] | select(.state | contains("APPROVED")) ] | length')
32+
test $CURRENT_APPROVALS_COUNT -ge 2 || exit 1 # At least 2 approvals are required
1633
- name: Launch workflow via Seqera Platform
1734
uses: seqeralabs/action-tower-launch@v2
1835
with:

‎.github/workflows/ci.yml

+87-17
Original file line numberDiff line numberDiff line change
@@ -7,50 +7,120 @@ on:
77
pull_request:
88
release:
99
types: [published]
10+
workflow_dispatch:
1011

1112
env:
13+
NFT_DIFF: "pdiff"
14+
NFT_DIFF_ARGS: "--line-numbers --width 120 --expand-tabs=2"
15+
NFT_VER: "0.9.0"
16+
NFT_WORKDIR: "~"
1217
NXF_ANSI_LOG: false
18+
NXF_SINGULARITY_CACHEDIR: ${{ github.workspace }}/.singularity
19+
NXF_SINGULARITY_LIBRARYDIR: ${{ github.workspace }}/.singularity
1320

1421
concurrency:
1522
group: "${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}"
1623
cancel-in-progress: true
1724

1825
jobs:
1926
test:
20-
name: Run pipeline with test data
27+
name: "Test ${{ matrix.filter }} | ${{ matrix.profile }} | ${{ matrix.NXF_VER }} | ${{ matrix.shard }}/5"
2128
# Only run on push if this is the nf-core dev branch (merged PRs)
2229
if: "${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/rnavar') }}"
2330
runs-on: ubuntu-latest
2431
strategy:
32+
fail-fast: false
2533
matrix:
2634
NXF_VER:
27-
- "23.04.0"
35+
- "24.04.2"
2836
- "latest-everything"
29-
test:
30-
- "default"
31-
- "annotation"
32-
- "removeduplicates"
33-
- "skipbasecalib"
34-
- "bamcsiindex"
37+
filter: ["pipeline"]
38+
# filter: ["process", "workflow", "function", "pipeline"]
39+
profile: ["conda", "docker", "singularity"]
40+
shard: [1, 2, 3, 4]
41+
isMaster:
42+
- ${{ github.base_ref == 'master' }}
43+
exclude:
44+
- isMaster: false
45+
profile: "conda"
46+
- isMaster: false
47+
profile: "singularity"
3548
steps:
3649
- name: Check out pipeline code
3750
uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4
51+
with:
52+
fetch-depth: 0
3853

39-
- name: Install Nextflow
54+
- name: Set up Nextflow
4055
uses: nf-core/setup-nextflow@v2
4156
with:
4257
version: "${{ matrix.NXF_VER }}"
4358

59+
- name: Set up nf-test
60+
uses: nf-core/setup-nf-test@v1
61+
with:
62+
version: ${{ env.NFT_VER }}
63+
64+
- name: Set up Apptainer
65+
if: matrix.profile == 'singularity'
66+
uses: eWaterCycle/setup-apptainer@main
67+
68+
- name: Set up Singularity
69+
if: matrix.profile == 'singularity'
70+
run: |
71+
mkdir -p $NXF_SINGULARITY_CACHEDIR
72+
mkdir -p $NXF_SINGULARITY_LIBRARYDIR
73+
74+
- name: Cache pdiff
75+
uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4
76+
id: cache-pip-pdiff
77+
with:
78+
path: ~/.cache/pip
79+
key: ${{ runner.os }}-pip-pdiff
80+
81+
- name: Set up pdiff to see diff between nf-test snapshots
82+
run: |
83+
python -m pip install --upgrade pip
84+
pip install pdiff cryptography
85+
86+
- name: Set up Miniconda
87+
if: matrix.profile == 'conda'
88+
uses: conda-incubator/setup-miniconda@a4260408e20b96e80095f42ff7f1a15b27dd94ca # v3
89+
with:
90+
miniconda-version: "latest"
91+
auto-update-conda: true
92+
conda-solver: libmamba
93+
channels: conda-forge,bioconda
94+
95+
- name: Set up Conda
96+
if: matrix.profile == 'conda'
97+
run: |
98+
echo $(realpath $CONDA)/condabin >> $GITHUB_PATH
99+
echo $(realpath python) >> $GITHUB_PATH
100+
44101
- name: Disk space cleanup
45102
uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
46103

47-
- name: Set up Python
48-
uses: actions/setup-python@v2
49-
with:
50-
python-version: "3.x"
104+
- name: "Run tests | ${{ matrix.filter }}_${{ matrix.profile }} | ${{ matrix.shard }}/5"
105+
run: |
106+
nf-test test \
107+
--ci \
108+
--debug \
109+
--verbose \
110+
--junitxml="TEST-${{ matrix.filter }}_${{ matrix.profile }}_${{ matrix.shard }}.xml" \
111+
--shard ${{ matrix.shard }}/5 \
112+
--changed-since HEAD^ \
113+
--follow-dependencies \
114+
--profile "+${{ matrix.profile }}" \
115+
--filter ${{ matrix.filter }}
51116
52-
- name: Install dependencies
53-
run: python -m pip install --upgrade pip pytest-workflow
117+
- name: Publish Test Report
118+
uses: mikepenz/action-junit-report@v4
119+
if: success() || failure() # always run even if the previous step fails
120+
with:
121+
report_paths: "TEST-*.xml"
54122

55-
- name: Run pipeline with tests settings
56-
run: TMPDIR=~ PROFILE=docker pytest --tag ${{ matrix.test }} --symlink --kwdof --git-aware --color=yes
123+
- name: Clean up
124+
if: always()
125+
run: |
126+
sudo rm -rf /home/ubuntu/tests/

‎.github/workflows/download_pipeline.yml

+43-10
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Test successful pipeline download with 'nf-core download'
1+
name: Test successful pipeline download with 'nf-core pipelines download'
22

33
# Run the workflow when:
44
# - dispatched manually
@@ -8,7 +8,7 @@ on:
88
workflow_dispatch:
99
inputs:
1010
testbranch:
11-
description: "The specific branch you wish to utilize for the test execution of nf-core download."
11+
description: "The specific branch you wish to utilize for the test execution of nf-core pipelines download."
1212
required: true
1313
default: "dev"
1414
pull_request:
@@ -39,9 +39,11 @@ jobs:
3939
with:
4040
python-version: "3.12"
4141
architecture: "x64"
42-
- uses: eWaterCycle/setup-singularity@931d4e31109e875b13309ae1d07c70ca8fbc8537 # v7
42+
43+
- name: Setup Apptainer
44+
uses: eWaterCycle/setup-apptainer@4bb22c52d4f63406c49e94c804632975787312b3 # v2.0.0
4345
with:
44-
singularity-version: 3.8.3
46+
apptainer-version: 1.3.4
4547

4648
- name: Install dependencies
4749
run: |
@@ -54,33 +56,64 @@ jobs:
5456
echo "REPOTITLE_LOWERCASE=$(basename ${GITHUB_REPOSITORY,,})" >> ${GITHUB_ENV}
5557
echo "REPO_BRANCH=${{ github.event.inputs.testbranch || 'dev' }}" >> ${GITHUB_ENV}
5658
59+
- name: Make a cache directory for the container images
60+
run: |
61+
mkdir -p ./singularity_container_images
62+
5763
- name: Download the pipeline
5864
env:
59-
NXF_SINGULARITY_CACHEDIR: ./
65+
NXF_SINGULARITY_CACHEDIR: ./singularity_container_images
6066
run: |
61-
nf-core download ${{ env.REPO_LOWERCASE }} \
67+
nf-core pipelines download ${{ env.REPO_LOWERCASE }} \
6268
--revision ${{ env.REPO_BRANCH }} \
6369
--outdir ./${{ env.REPOTITLE_LOWERCASE }} \
6470
--compress "none" \
6571
--container-system 'singularity' \
66-
--container-library "quay.io" -l "docker.io" -l "ghcr.io" \
72+
--container-library "quay.io" -l "docker.io" -l "community.wave.seqera.io" \
6773
--container-cache-utilisation 'amend' \
68-
--download-configuration
74+
--download-configuration 'yes'
6975
7076
- name: Inspect download
7177
run: tree ./${{ env.REPOTITLE_LOWERCASE }}
7278

79+
- name: Count the downloaded number of container images
80+
id: count_initial
81+
run: |
82+
image_count=$(ls -1 ./singularity_container_images | wc -l | xargs)
83+
echo "Initial container image count: $image_count"
84+
echo "IMAGE_COUNT_INITIAL=$image_count" >> ${GITHUB_ENV}
85+
7386
- name: Run the downloaded pipeline (stub)
7487
id: stub_run_pipeline
7588
continue-on-error: true
7689
env:
77-
NXF_SINGULARITY_CACHEDIR: ./
90+
NXF_SINGULARITY_CACHEDIR: ./singularity_container_images
7891
NXF_SINGULARITY_HOME_MOUNT: true
7992
run: nextflow run ./${{ env.REPOTITLE_LOWERCASE }}/$( sed 's/\W/_/g' <<< ${{ env.REPO_BRANCH }}) -stub -profile test,singularity --outdir ./results
8093
- name: Run the downloaded pipeline (stub run not supported)
8194
id: run_pipeline
8295
if: ${{ job.steps.stub_run_pipeline.status == failure() }}
8396
env:
84-
NXF_SINGULARITY_CACHEDIR: ./
97+
NXF_SINGULARITY_CACHEDIR: ./singularity_container_images
8598
NXF_SINGULARITY_HOME_MOUNT: true
8699
run: nextflow run ./${{ env.REPOTITLE_LOWERCASE }}/$( sed 's/\W/_/g' <<< ${{ env.REPO_BRANCH }}) -profile test,singularity --outdir ./results
100+
101+
- name: Count the downloaded number of container images
102+
id: count_afterwards
103+
run: |
104+
image_count=$(ls -1 ./singularity_container_images | wc -l | xargs)
105+
echo "Post-pipeline run container image count: $image_count"
106+
echo "IMAGE_COUNT_AFTER=$image_count" >> ${GITHUB_ENV}
107+
108+
- name: Compare container image counts
109+
run: |
110+
if [ "${{ env.IMAGE_COUNT_INITIAL }}" -ne "${{ env.IMAGE_COUNT_AFTER }}" ]; then
111+
initial_count=${{ env.IMAGE_COUNT_INITIAL }}
112+
final_count=${{ env.IMAGE_COUNT_AFTER }}
113+
difference=$((final_count - initial_count))
114+
echo "$difference additional container images were \n downloaded at runtime . The pipeline has no support for offline runs!"
115+
tree ./singularity_container_images
116+
exit 1
117+
else
118+
echo "The pipeline can be downloaded successfully!"
119+
fi

‎.github/workflows/linting.yml

+19-4
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
name: nf-core linting
22
# This workflow is triggered on pushes and PRs to the repository.
3-
# It runs the `nf-core lint` and markdown lint tests to ensure
3+
# It runs the `nf-core pipelines lint` and markdown lint tests to ensure
44
# that the code meets the nf-core guidelines.
55
on:
66
push:
@@ -41,17 +41,32 @@ jobs:
4141
python-version: "3.12"
4242
architecture: "x64"
4343

44+
- name: read .nf-core.yml
45+
uses: pietrobolcato/[email protected]
46+
id: read_yml
47+
with:
48+
config: ${{ github.workspace }}/.nf-core.yml
49+
4450
- name: Install dependencies
4551
run: |
4652
python -m pip install --upgrade pip
47-
pip install nf-core
53+
pip install nf-core==${{ steps.read_yml.outputs['nf_core_version'] }}
54+
55+
- name: Run nf-core pipelines lint
56+
if: ${{ github.base_ref != 'master' }}
57+
env:
58+
GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }}
59+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
60+
GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }}
61+
run: nf-core -l lint_log.txt pipelines lint --dir ${GITHUB_WORKSPACE} --markdown lint_results.md
4862

49-
- name: Run nf-core lint
63+
- name: Run nf-core pipelines lint --release
64+
if: ${{ github.base_ref == 'master' }}
5065
env:
5166
GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }}
5267
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
5368
GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }}
54-
run: nf-core -l lint_log.txt lint --dir ${GITHUB_WORKSPACE} --markdown lint_results.md
69+
run: nf-core -l lint_log.txt pipelines lint --release --dir ${GITHUB_WORKSPACE} --markdown lint_results.md
5570

5671
- name: Save PR number
5772
if: ${{ always() }}

‎.github/workflows/linting_comment.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ jobs:
1111
runs-on: ubuntu-latest
1212
steps:
1313
- name: Download lint results
14-
uses: dawidd6/action-download-artifact@09f2f74827fd3a8607589e5ad7f9398816f540fe # v3
14+
uses: dawidd6/action-download-artifact@bf251b5aa9c2f7eeb574a96ee720e24f801b7c11 # v6
1515
with:
1616
workflow: linting.yml
1717
workflow_conclusion: completed

‎.github/workflows/release-announcements.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ jobs:
1212
- name: get topics and convert to hashtags
1313
id: get_topics
1414
run: |
15-
echo "topics=$(curl -s https://nf-co.re/pipelines.json | jq -r '.remote_workflows[] | select(.full_name == "${{ github.repository }}") | .topics[]' | awk '{print "#"$0}' | tr '\n' ' ')" >> $GITHUB_OUTPUT
15+
echo "topics=$(curl -s https://nf-co.re/pipelines.json | jq -r '.remote_workflows[] | select(.full_name == "${{ github.repository }}") | .topics[]' | awk '{print "#"$0}' | tr '\n' ' ')" | sed 's/-//g' >> $GITHUB_OUTPUT
1616
1717
- uses: rzr/fediverse-action@master
1818
with:

0 commit comments

Comments
 (0)