BIDS filter doesnt work as expected for qsiprep pipeline

[bilas003]

Summary

BIDS filter doesnt work as expected for qsiprep pipeline

Additional details

• QSIPrep version:0.20.1.dev0+geda84d5.d2024011
• Singularity version: qsiprep_0.20.0.sif

What were you trying to do?

I have a bids filter designed to filter out the desired scans in the qsiprep pipeline

What did you expect to happen?

The output should only contain the files mentioned in the BIDS filter

What actually happened?

Contains files other than mentioned in the filter

Reproducing the bug

To test if the filter is correct, I ran the filter in PYbids and the output is as expected but fails while incorporated with qsiprep pipeline.

[smeisler]

Hi @bilas003,

May you please see if the error persists with the most recent version (pennlinc dockerhub, 1.0.0rc1)?

[bilas003]

Thank you Steve! I am using the latest build qsiprep-1.0.0rc1.sif, is --skip-bids-validation not permitted for this version?

error received is : qsirecon: error: unrecognized arguments: --skip-bids-validation

[smeisler]

Sounds like you're running Qsirecon, not qsiprep, for which that flag is not permitted.

[bilas003]

Thank you ! corrected it.

[bilas003]

Hi Steve,

The job with Bids-filter just completed without error with new release. The BIDS filter criteria is not working on the files.

[smeisler]

If you're running Qsirecon, have you made sure the filter is relevant to the qsiprep input directory (as opposed to raw)?

[bilas003]

I ran with qsiprep (qsiprep-1.0.0.sif) this time.

[smeisler]

May you please share all the pertinent details of the command, the filter, and tree structure of a subject's raw data?

[bilas003]

BIDS-Filter json defined

{
"sbref":{"datatype":"dwi","suffix":"sbref","session": "V02"},
"t2w": {"datatype": "anat","suffix":"T2w", "run":"2","session":"V02","SeriesDescription":"anat-t2w","ProtocolName":"anat-t2w","acquisition":null},
"dwi":{"datatype":"dwi","suffix":"dwi","session": "V02"}

}

Raw data:

Command used to run the pipeline:

Please let me know if you can see the screenshots properly.

[smeisler]

What does a BIDS validation report return?

[smeisler]

Also, it is better to put code as formatted text, using tickmarks so it looks nice

like this

Additionally, things like "SeriesDescription":"anat-t2w","ProtocolName":"anat-t2w" do not belong in a filter file.

[smeisler]

I also worry your filter file is not allowing fmaps in.

[bilas003]

Hi Steve,

I changed bids filter file , it looks like this now

{
"fmap": {"datatype": "fmap"},
"sbref":{"datatype":"dwi","suffix":"sbref","session": "V02"},
"t2w": {"datatype": "anat","suffix":"T2w", "run":"2","session":"V02","acquisition":null},
"dwi":{"datatype":"dwi","suffix":"dwi","session": "V02"}
}

The job ran all fine without error, attaching the output log. The final filtered dataset is not as expected.

******************* log *********************************************************************************************
250127-15:35:15,491 nipype.workflow IMPORTANT:
Running QSIPrep version 1.0.1.dev0+gee9aa2e.d20250115

     License NOTICE ##################################################
     QSIPrep 1.0.1.dev0+gee9aa2e.d20250115
     Copyright The PennLINC Developers.

     This product includes software developed by
     the NiPreps Community (https://nipreps.org/).

     Portions of this software were developed at the Department of
     Psychology at Stanford University, Stanford, CA, US.

     This software is also distributed as a Docker container image.
     The bootstrapping file for the image ("Dockerfile") is licensed
     under the MIT License.

     This software may be distributed through an add-on package called
     "Docker Wrapper" that is under the BSD 3-clause License.
     #################################################################

250127-15:35:30,61 nipype.workflow IMPORTANT:
Building QSIPrep's workflow:
* BIDS dataset path: /data.
* Participant list: ['XXXXX'].
* Run identifier: 20250127-152900_82000810-ebef-40fa-945c-c3e1510446c7.
250127-15:35:30,487 nipype.workflow WARNING:
BIDS filter file value for session may conflict with values specified on the command line
250127-15:35:30,487 nipype.workflow WARNING:
BIDS filter file value for session may conflict with values specified on the command line
250127-15:35:30,487 nipype.workflow WARNING:
BIDS filter file value for session may conflict with values specified on the command line
250127-15:35:30,581 nipype.workflow IMPORTANT:
Collected data:
dwi:

• /data/sub-XXXXX/ses-V02/dwi/sub-XXXXX_ses-V02_dir-AP_run-1_dwi.nii.gz
• /data/sub-XXXXX/ses-V02/dwi/sub-XXXXX_ses-V02_dir-PA_run-1_dwi.nii.gz
  flair: []
  fmap:
• /data/sub-XXXXX/ses-V02/fmap/sub-XXXXX_ses-V02_dir-AP_run-1_epi.nii.gz
• /data/sub-XXXXX/ses-V02/fmap/sub-XXXXX_ses-V02_dir-AP_run-2_epi.nii.gz
• /data/sub-XXXXX/ses-V02/fmap/sub-XXXXX_ses-V02_dir-AP_run-3_epi.nii.gz
• /data/sub-XXXXX/ses-V02/fmap/sub-XXXXX_ses-V02_dir-PA_run-1_epi.nii.gz
• /data/sub-XXXXX/ses-V02/fmap/sub-XXXXX_ses-V02_dir-PA_run-2_epi.nii.gz
• /data/sub-XXXXX/ses-V03/fmap/sub-XXXXX_ses-V03_acq-anat_run-1_TB1TFL.nii.gz
• /data/sub-XXXXX/ses-V03/fmap/sub-XXXXX_ses-V03_acq-anat_run-2_TB1TFL.nii.gz
• /data/sub-XXXXX/ses-V03/fmap/sub-XXXXX_ses-V03_acq-famp_run-1_TB1TFL.nii.gz
• /data/sub-XXXXX/ses-V03/fmap/sub-XXXXX_ses-V03_acq-famp_run-2_TB1TFL.nii.gz
• /data/sub-XXXXX/ses-V03/fmap/sub-XXXXX_ses-V03_dir-AP_run-1_epi.nii.gz
• /data/sub-XXXXX/ses-V03/fmap/sub-XXXXX_ses-V03_dir-AP_run-2_epi.nii.gz
• /data/sub-XXXXX/ses-V03/fmap/sub-XXXXX_ses-V03_dir-PA_run-1_epi.nii.gz
• /data/sub-XXXXX/ses-V03/fmap/sub-XXXXX_ses-V03_dir-PA_run-2_epi.nii.gz
  roi: []
  sbref:
• /data/sub-XXXXX/ses-V02/dwi/sub-XXXXX_ses-V02_dir-AP_run-1_sbref.nii.gz
• /data/sub-XXXXX/ses-V02/dwi/sub-XXXXX_ses-V02_dir-PA_run-1_sbref.nii.gz
  t1w:
• /data/sub-XXXXX/ses-V03/anat/sub-XXXXX_ses-V03_run-1_T1w.nii.gz
  t2w:
• /data/sub-XXXXX/ses-V02/anat/sub-XXXXX_ses-V02_run-2_T2w.nii.gz

250127-15:35:30,597 nipype.workflow INFO:
Running nonlinear normalization to template
250127-15:35:30,633 nipype.workflow INFO:
Combining all DWI files within each available session and acquisition:
250127-15:35:30,636 nipype.workflow INFO:
- 2 scans in session V02/acquisition None
250127-15:35:30,694 nipype.workflow INFO:
[{'dwi_series': ['/data/sub-XXXXX/ses-V02/dwi/sub-XXXXX_ses-V02_dir-PA_run-1_dwi.nii.gz'], 'dwi_series_pedir': 'j', 'fieldmap_info': {'suffix': 'rpe_series', 'rpe_series': ['/data/sub-XXXXX/ses-V02/dwi/sub-XXXXX_ses-V02_dir-AP_run-1_dwi.nii.gz']}, 'concatenated_bids_name': 'sub-XXXXX_ses-V02_run-1'}]
250127-15:35:31,175 nipype.workflow INFO:
Automatically using 3, 3, 3 window for dwidenoise
250127-15:35:31,314 nipype.workflow INFO:
Automatically using 3, 3, 3 window for dwidenoise
250127-15:35:31,328 nipype.workflow INFO:
Using 8 threads in eddy
250127-15:35:31,344 nipype.workflow INFO:
Using single-stage SDC, TOPUP-only
250127-15:35:35,589 nipype.workflow INFO:
QSIPrep workflow graph with 280 nodes built successfully.
250127-15:35:57,458 nipype.workflow IMPORTANT:
QSIPrep started!
250127-15:36:24,484 nipype.interface WARNING:
Changing /out/sub-XXXXX/anat/sub-XXXXX_space-ACPC_desc-brain_mask.nii.gz dtype from float64 to uint8
250127-15:36:34,330 nipype.interface WARNING:
Changing /out/sub-XXXXX/anat/sub-XXXXX_space-ACPC_desc-aseg_dseg.nii.gz dtype from float64 to int16
250127-15:36:38,70 nipype.interface WARNING:
Changing /out/sub-XXXXX/anat/sub-XXXXX_space-ACPC_dseg.nii.gz dtype from uint32 to int16
250127-15:38:07,988 nipype.interface WARNING:
Changing /out/sub-XXXXX/ses-V02/dwi/sub-XXXXX_ses-V02_space-ACPC_desc-brain_mask.nii.gz dtype from float64 to uint8
250127-15:38:13,693 nipype.workflow IMPORTANT:
QSIPrep finished successfully!
250127-15:38:13,701 nipype.workflow IMPORTANT:
Works derived from this QSIPrep execution should include the boilerplate text found in <OUTPUT_PATH>/logs/CITATION.md.

---

[smeisler]

Looks like the"run" entity is supposed to be an integer type, not string, according to here: https://github.com/bids-standard/pybids/blob/maint/0.17.x/bids/layout/config/bids.json#L62. I also wonder if some of your problems wouldn't be solved by specifying the --session-id and --anat-modality. if that doesn't work, its always a possibility to symlink just the necessary data to temporary filespaces a create mini fake bids directories. The tickmark trick I specified in my earlier comment is also applicable to terminal outputs (such as the tree structure and qsiprep output log).

[erikglee]

Hi Steve, thanks for your replies! Right now we run processing using a similar routine to what you are describing, but we are hoping to switch over to relying more on BIDS filter files so that it is easier for other users to replicate our processing results. We had not picked up on that detail about the 'run' entity being an integer, so we will give that a try and report back.

[bilas003]

update : run entity changed to integer
Ran the pipeline and its complete with zero error. The output file shows no change.