Skip to content

OpenPipelines.bio v1.0.0
Compare
Choose a tag to compare
@DriesSchaumont DriesSchaumont released this 12 Jun 12:14
· 11 commits to release since this release
1.0.0
a7f8635

BREAKING CHANGES

  • query/cellxgene_census: Refactored the interface, documentation and internal workings of this component (PR #621).

    • Renamed arguments to align with standard OpenPipelines notations and cellxgene census API:
      • --input_database became --input_uri
      • --cellxgene_release became --census_version
      • --cell_query became --obs_value_filter
      • --cells_filter_columns became --cell_filter_grouping
      • --min_cells_filter_columns became --cell_filter_minimum_count
      • --modality became --output_modality
      • Removed --dataset_id since it was no longer being used.
      • Added --add_dataset_meta to add metadata to the output MuData object.
    • Documentation of the component and its arguments was improved.

  • Docker image names now use / instead of _ between the name of the component and the namespace (PR #712).

  • Change separator for arguments with multiple inputs from : to ; (PR #700 and #707). Now, all arguments with multiple: true will use ; as the separator.
    This change was made to be able to deal with file paths that contain :, e.g. s3://my-bucket/my:file.txt. Furthermore, the ; separator will become
    the default separator for all arguments with multiple: true in Viash >= 0.9.0.

  • This project now uses viash version 0.8.4 to build components and workflows. Changes related to this version update should
    be mostly backwards compatible with respect to the results and execution of the pipelines. From a development perspective,
    drastic updates have been made to the developemt workflow.

    Development related changes:

    • Bump viash version to 0.8.4 (PR #598, PR#638, #697 and #706) in the project configuration.
    • All pipelines no longer use the anonymous workflow. Instead, these workflows were given
      a name which was added to the viash config as the entrypoint to the pipeline (PR #598).
    • Removed the workflows folder and moved its contents to new locations:

      1. The resources_test_scripts folder now resides in the root of the project (PR #605).

      2. All workflows have been moved to the src/workflows folder (PR #605).
        This implies that workflows must now be build using viash (ns) build, just like with components.

      3. Adjust GitHub Actions to account for new workflow paths (PR #605).

      4. In order to be backwards compatible, the workflows folder now contains symbolic
        links to the build workflows in target. This is not a problem when using the repository for pipeline
        execution. However, if a developer wishes to contribute to the project, symlink support should be enabled
        in git using git config core.symlinks=true. Alternatively, use
        git clone -c core.symlinks=true [email protected]:openpipelines-bio/openpipeline.git when cloning the
        repository. This avoids the symlinks being resolved (PR #628).
        4bis. With PR #668, the workflows have been renamed. This does not hamper the backwards compatibility
        of the symlinks that have been described in 4, because they still use the original location
        which includes the original name.
        * multiomics/rna_singlesample has been renamed to rna/process_single_sample,
        * multiomics/rna_multisample has been renamed to rna/rna_multisample,
        * multiomics/prot_multisample became prot/prot_multisample,
        * multiomics/prot_singlesample became prot/prot_singlesample,
        * multiomics/full_pipeline was moved to multiomics/process_samples,
        * multiomics/multisample has been renamed to multiomics/process_batches,
        * multiomics/integration/initialize_integration changed to multiomics/dimensionality_reduction,
        * finally, all workflows at multiomics/integration/* were moved to integration/*

      5. Removed the workflows/utils folder. Functionality that was provided by the DataflowHelper
        and WorkflowHelper is now being provided by viash when the workflow is being build (PR #605).

    End-user facing changes:

    • The concat component had been deprecated and will be removed in a future release.
      It's functionality has been copied to the concatenate_h5mu component because the name is in
      conflict with the concat operator from nextflow (PR #598).
    • prot_singlesample, rna_singlesample, prot_multisample and rna_multisample: QC statistics
      are now only calculated once where needed. This means that the mitochondrial gene detection is
      performed in the rna_singlesample pipeline and the other count based statistics are calculated
      during the prot_multisample and rna_multisample pipelines. In both cases, the qc pipeline
      is being used, but only parts of that workflow are activated by parametrization. Previously
      the count based statistics were calculated in both the singlesample and multisample pipelines,
      with the results from the multisample pipelines overwriting the previous results. What is breaking here
      is that the qc statistics are not being added to the results of the singlesample worklows.
      This is not an issue when using the full_pipeline because in this case the singlesample and
      multisample workflows are executed in-tandem. If you wish to execute the singlesample workflows
      in a seperate manner and still include count based statistics, please run the qc pipeline
      on the result of the singlesample workflow (PR #604).
    • filter/filter_with_hvg has been renamed to feature_annotation/highly_variable_features_scanpy, along with the following changes (PR #667).
      • --do_filter was removed
      • --n_top_genes has been renamed to --n_top_features
    • full_pipeline, multisample and rna_multisample: Renamed arguments (PR #667).
      • --filter_with_hvg_var_output became --highly_variable_features_obs_batch_key
      • --filter_with_hvg_obs_batch_key became --highly_variable_features_var_output
    • rna_multisample: Renamed arguments (PR #667).
      • --filter_with_hvg_n_top_genes became --highly_variable_features_n_top_features
      • --filter_with_hvg_flavor became --highly_variable_features_flavor

  • Renamed obsm_metrics to uns_metrics for the cellranger_mapping workflow because the cellranger metrics are stored in .uns and not .obsm (PR #610).

  • mapping/cellranger_mkfastq: update from cellranger 6.0.2 to 7.0.1 (PR #675)

BUG FIXES

  • mapping/cellranger_multi: Fix the regex for the fastq input files to allow dropping the lane from the input file names (e.g. _L001) (PR #778).

  • workflows/rna/rna_singlesample: Fix argument passing top_n_vars and obs_name_mitochondrial_fraction to the qc subworkflow (PR #779).

  • rna_singlesample: fixed a bug where selecting the column for the filtering with mitochondrial fractions
    using obs_name_mitochondrial_fraction was done with the wrong column name, causing ValueError (PR #743).

  • Fix publishing in process_samples and process_batches (PR #759).

  • Cellranger multi: Fix using a relative input path for --vdj_inner_enrichment_primers (PR #717)

  • dataflow/split_modalities: remove unused compression argument. Use output_compression instead (PR #714).

  • metadata/grep_annotation_column: fix calculating fraction when an input observation has no counts, which caused
    the result to be out of bounds.

  • Fix --output argument not working for several workflows (PR #740).

  • transform/log1p: fix --input_layer argument not functioning (PR #678).

  • dataflow/concat and dataflow/concatenate_h5mu: Fix an issue where using --mode move on samples with non-overlapping features would cause var_names to become unaligned to the data (PR #653).

  • filter/filter_with_scrublet: (Testing) Fix duplicate test function names (PR #641).

  • dataflow/concatenate_h5mu and dataflow/concat: Fix TypeError when using mode 'move' and a column with conflicting metadata does not exist across all samples (PR #631).

  • dataflow/concatenate_h5mu and dataflow/concat: Fix an issue where joining columns with different datatypes caused TypeError (PR #619).

  • qc/calculate_qc_metrics: Resolved an issue where statistics based on the input columns selected with --var_qc_metrics were incorrect when these input columns were encoded in pd.BooleanDtype() (PR #685).

  • move_obsm_to_obs: fix setting output columns when they already exist (PR #690).

  • workflows/dimensionality_reduction workflow: nearest neighbour calculations no longer recalcalates the PCA when obm_input --obsm_pca is not set to X_pca.

  • feature_annotation/highly_variable_scanpy: fix .X being used to remove observations with 0 counts when --layer has been specified.

  • filter/filter_with_counts: fix --layer argument not being used.

  • transform/normalize_total: fix incorrect layer being written to the output when the input layer if not .X.

  • src/workflows/qc: fix input layer not being taken into account when calculating the fraction of mitochondrial genes (always used .X).

  • convert/from_cellranger_multi_to_h5mu: fix metric values not repesented as percentages being devided by 100. (#704).

NEW FUNCTIONALITY

  • dimred/tsne component: Added a tSNE dimensionality reduction component (PR #742).

  • multisample pipeline: This workflow now works when provided multimple unimodal files or multiple multimodal files, in addition to the previously supported single multimodal file (PR #606). The modalities are processed independently from each other:

    • As before, a single multimodal file is split into several unimodal MuData objects, each modality being stored in a file.
    • (New) When multiple unimodal files are provided, they can be used used as is.
    • (New) Mosaic input (i.e. multiple uni- or multimodal files) are split into unimodal files.
      Providing the same modality twice is not supported however, meaning the modalities should be unique.
      For example, using input: ["data1.h5mu", "data2.h5mu"] with data1.h5mu providing data for rna and atac
      and data2.h5mu for rna and prot will not work, because the rna modality is present in both input files.

  • multisample workflow: throw an error when argument values for the merge component or the initialize_integration workflow differ between the inputs (PR #606).

  • Added a split_modalities workflow in order to split a multimodal mudata files into several unimodal mudata files. Its behavior is identical to the split_modalities component, but it also provides functionality to make sure everything works when nextflow's -stub option is enabled (PR #606).

  • All workflow now use dependencies to handle includes from other workflows (PR #606).

  • qc/calculate_qc_metrics: allow setting the output column names and disabling the calculation of several metrics (PR #644).

  • rna_multisample, prot_multisample and qc workflows: allow setting the output column names and disabling the calculation of several metrics (PR #606).

  • cluster/leiden: Allow calculating multiple resolutions in parallel (PR #645).

  • qc/calculate_qc_metrics: allow setting the output column names and disabling the calculation of several metrics (PR #644).

  • rna_multisample workflow: added --modality argument (PR #607).

  • multisample workflow: in addition to using multimodal files as input, this workflow now also accepts a list of files. The list of files must be the unimodal equivalents of a split multimodal file. The modalities in the list must be unique and after processing the modalities will be merged into multimodal files (PR #606).

  • Added filter/intersect_obs component which removes observations that are not shared between modalities (PR #589).

  • Re-enable convert/from_h5mu_to_seurat component (PR #616).

  • Added the gdo_singlesample pipeline with basic count filtering (PR #672).

  • process_samples pipeline: the --rna_layer, --prot_layer and gdo_layer argument can not be used to specify an alternative layer to .X where the raw data are stored. To enable this feature, the following changes were required:

    • Added transform/move_layer component.
    • filter/filter_with_scrublet: added --layer argument.
    • transform/clr: added --input_layer argument.
    • metadata/grep_annotation_column: added --input_layer argument.
    • rna/rna_singlesample, rna/rna_multisample, prot/prot_singlesample and prot/prot_multisample: add --layer argument.
    • process_batches: Added rna_layer and prot_layer arguments.

  • Enable dataset functionality for nf-tower (PR #701)

  • Added annotate/score_genes and annotate/score_genes_cell_cycle to calculate scanpy gene scores (PR #703).

MINOR CHANGES

  • metadata/grep_annotation_column: Added more logging output (PR #697).

  • metadata/add_id and metadata/grep_annotation_column: Bump python to 3.11 (PR #697).

  • dataflow/split_modalities: add more logging output and bump python to 3.12 (PR #714).

  • correction/cellbender: Update nextflow resource labels from singlecpu and lowmem to midcpu and midmem (PR #736)

  • Refactored rna_multisample (PR #607), cellranger_multi (PR #609), cellranger_mapping (PR #610) and other (PR #606) pipelines to use fromState and toState functionality.

  • metadata/add_id: add more runtime logging (PR #663).

  • cluster/leiden: Bump python to 3.11 and leidenalg to 0.10.0 (PR #645).

  • mapping/htseq_count_to_h5mu and multi_star: update polars and gtfparse (PR #642).

  • Pin from_h5mu_to_seurat to use Seurat to version 4 (PR #630).

  • velocity/scvelo: bump scvelo to 0.3.1 and python to 3.10 (PR #640).

  • Updated the Viash YAML schemas to the latest version of Viash (PR #620).

  • build_cellranger_reference and build_bdrhap_reference: Bump go version to 1.21.4 when building seqkit for testing the component (PR #624 and PR #637).

  • correction/cellbender_remove_background: Remove muon as a test dependency (PR #636).

  • (Automatic testing) Update viashpy to 0.6.0 (PR #665).

  • integrate/scarches, integrate/scvi, velocity/scvelo and integrate/totalvi: pin jax, jaxlib to <0.4.23 (PR #699).

  • integrate/scvi: Unpin numba and pin scvi-tools to 1.0.3 (PR #699).

  • integrate/totalvi: Enable GPU-accelerated computing, unpin torchmetrics and pin jax, jaxlib to <0.4.23 (PR #699).

Assets 2
Loading