RefGenie integration

[ewels]

RefGenie is a very nice command-line tool to manage reference genomes locally: http://refgenie.databio.org/en/latest/

The Python package can be imported by other tools so we could potentially make a nf-core refgenie subcommand to fetch whatever references the user has installed and build a nextflow / nf-core config file with these paths. This could be saved / linked to from ~/.nextflow/conf so that the user has these available for any pipeline runs (or a custom supplied path).

Note that we may need to update nf-core pipelines to use the same reference type identifiers as RefGenie to work, or at least provide a translation table. e.g. bismark > bismark_bt2_index

[ewels]

@nsheff managed to add plugin hooks and build an example @nf-core plugin all in a day! Over here: https://github.com/databio/refgenie_nfcore

Can use this to model the RefGenie integration. This will work super nicely because it is so harmless:

• RefGenie installed, nf-core not installed: Nothing
• RefGenie not installed, nf-core installed: Nothing
• RefGenie and nf-core installed: Nextflow genomes config automagically maintained

There will be no action required by users beyond installing two tools that they will be using anyway. 🚀

Phil

[ggabernet]

Would this really be part of the 1.10 milestone?

[ewels]

I was hoping to get it there but maybe we shift it back to the next release.

[stevekm]

@ewels was there more progress needed on this?

[nsheff]

anything I can do to help?

[nsheff]

I wanted to announce an update that would allow using refgenie for unarchived cloud assets directly. You first need a refgenie digest for the genome of interest, which you can get at an endpoint like this: http://rg.databio.org/genomes/genome_digest/hg38.

Then, you can use that with the assets/file_path endpoint to return either an http or s3 URL to the file of interest. For example:

• http://rg.databio.org/assets/file_path/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/fasta/fasta?remoteClass=s3
• http://rg.databio.org/assets/file_path/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/fasta/fasta?remoteClass=http

you can also use individual seek keys, just like the CLI, to get individual items within an asset:
http://rg.databio.org/assets/file_path/2230c535660fb4774114bfa966a62f823fdb6d21acf138d4/fasta/chrom_sizes?remoteClass=s3

One way to auto-generate a config file is to use the new refgenie populate function.

you would create a template like:

params {
  // illumina iGenomes reference file paths
  genomes {
    'GRCh37' {
      fasta       = "refgenie://hg38/fasta"
      bwa         = "refgenie://hg38/bwa_index"
      bowtie2     = "refgenie://hg38/bowtie2_index"
      }
    }
}

then you just run some flavor of refgenie populate file.tpl and you'd get the above, using the then-current URIs

[ewels]

I think that we really have 3 separate tasks with RefGenie now, so I will split them out into separate issues and close this one.

[ewels]

Moved into #1084 #1085 and #1086