Parameter Validation

[apeltzer]

Hi team,

as there was noone taking up the baton from this Thread here https://twitter.com/tallphil/status/1360938099925213184 I tried to give it a start and maybe others can chime in.

nf-core recently started with work mostly done by @KevinMenden , @drpatelh , @ewels and others to have validation of input parameters for pipelines. They created a way to validate parameters supplied to a pipeline against the already established JSON schema that nf-core pipelines inherently use for creating the launch feature on the webpage.

An example including it into the latest nf-core template on the dev branch can be found here:

nf-core/tools#852

Paolo suggested that this could be something that Nextflow itself could potentially support as a core feature. There are several points that this currently does:

• Validation of supplied parameters against the ones specified in the JSON schema
• Raising warnings if users specify parameters that are NOT in the JSON schema
• Raising warnings if users specify parameters incorrectly that are core nextflow parameters (e.g. having a --profile , though I'm not sure what happens with a --profile foobar -profile test,docker 🤔 )
• Generating automatic help & summary pages when users specify --help from the JSON schema (=> no need to keep this manually in sync, much less hardcoded parts in the pipeline & works generically between pipelines)

We also have further functionality in there, but these seem to be rather nf-core specific (e.g. the nf-core headers, e-mail feature are also in separate lib/ code pieces), so maybe not as suitable as the above to be of potential interest for core nextflow features.

Just wanted to give this a kick off and bring in the people really involved, I was just a fortunate front-line & happy prototype user 🤗

[pditommaso]

Indeed, I think it having this feature supported natively by NF would bring some important benefits:

• make validation avail to any NF pipeline which includes the scheme definition file.
• decouple the validation logic from pipeline implementation and reduce the boilerplate
• allow re-reusing the same validation logic in web frontend eg tower or nf-core launch

My understanding is that the code is implemented in Groovy, therefore it should be straightforward to have it supported by the NF runtime.

My only concern is that you may need to iterate fast with validation feature to add new features or improve functionalities. etc. This is why I think it could be useful to have the validation as nextflow plugin. By doing that, the validation plugin could have its own release cycle without having to wait for the NF release cadence. Thoughts?

[abhi18av]

+1 for implementing the validation as a plugin. The validation schema might evolve in future with Nextflow features and having it in the core might add dependencies to the dev cycle.

[apeltzer]

Sounds indeed like a reasonable way to approach this - lets see what others also think about this :-)

[vsmalladi]

That would be nice. For the json-schema would be able to move forward with new json-schema version support.

[ewels]

Awesome stuff, thanks @apeltzer! I'll try to fill in some details here and some pitfalls to be aware of in what we've done so far.

History

Original kudos actually goes back to @sven1103 and @msmootsgi way back in #866 - we were starting to get carried away with having funky comment strings in nextflow config files and other custom formats and @msmootsgi pushed for us to use JSON Schema instead. Great call, saved us from creating another new standard and has worked beautifully 👍🏻

For those unfamiliar with it, JSON Schema is a standardised format / vocabulary for writing a document (schema) that describes what a piece of structured data should look like. Official site is here: https://json-schema.org/ and excellent documentation is here: https://json-schema.org/understanding-json-schema/

Initially we started playing with writing a metaschema - that is, a schema that describes how our schema should be created, which in turn describe how the input data should look. After a long pause and then lot of head scratching I took a step back and decided that this was overkill and returned to vanilla JSON schema documents (almost, see below) using the Draft-07 spec (the latest when I wrote this code).

The @nf-core pipeline template now comes with a file called nextflow_schema.json in the repo root which is a skeleton schema corresponding to the parameters in the skeleton template pipeline.

The schema builder

Writing these big and complex JSON Schema documents by hand is no fun, and asking all pipeline developers to do it without making mistakes was never going to work. So the first thing that we did was to build a tool for the developers to build the schema. This is used by running the command nf-core schema build on a pipeline directory to either create a new schema file from scratch or update one that already exists. Broadly speaking, it does the following:

• If there is already a schema file present, validate it. If not, start with the one from the pipeline template.
• Call nextflow config and pull out anything starting with params. - ignore anything with a nested option
• Prompt the user to add any missing params to the schema
• Prompt the user to remove any schema params not found in the pipeline
• Launch https://nf-co.re/pipeline_schema_builder with a POST request containing the schema
• The website caches the schema and returns a cache ID to the command line. The CLI now waits indefinitely, checking the website for that ID until it's status is changed to complete.
• The user uses the graphical drag-and-drop interface to organise parameters - order, grouping, type, description, help text, logo and additional validation params (required, regex etc).
• When complete, the user hits save and the status is updated. The CLI notices, pulls the updated schema and saves to disk.

This has worked really well so far and we now have a lot of pipelines with complete and valid schema files, where no-one has had to touch a line of JSON by hand.

Compromises and weird stuff

This builder code went through a lot of iterations and changes. We ended up with a few strange things:

• Parameters are grouped by using JSON Schema definition elements, stitched together using allOf
  • We started with regular schema objects which can contain other schema items but then this messes up data validation as the pipeline parameters are not grouped in this way. Using definition elements in this way allows some formatting sugar without affecting how the schema functions at all. This came at quite a late stage as recommended by JSON Schema overlord @Relequestual 🙇🏻 and required a substantial rewrite, but was totally worth it as it keeps everything downstream much simpler.
• We have some custom elements
  • Proper vanilla JSON Schema don't have fields called fa_icon or help_text. We added these to allow rich documentation interfaces which was one of our main goals. Technically we could now write our own nf-core metaschema to state that our schema documents should have these keys and that they should work in a certain way, but I haven't found a reason to do this yet and it seems to work fine and not interfere with any usage that I've seen.
• Flat config only
  • This is probably the main one. As it stands, the schema builder and all downstream tools that we have built simply ignore any nextflow parameter that is nested (eg. params.foo.bar = 'baz'). The interface code was already getting super complex and as @nf-core pipelines generally don't use this style of config (with one or two exceptions) we didn't need it. I have in my head that nested params like this are difficult / impossible to specify on the command line (eg. --foo.bar = baz) but I might be wrong about that.
  • The JSON Schema format itself has no problem with nested parameters. A parameter can have type object` which then allows it to contain other parameters. We sort of semi-planned this for future expansion as I didn't love it at the time, but again this schema setup will be ignored by all of our downstream tools (hopefully it won't break them, but I haven't tried).

The schema launcher

Once we had a builder and some pipeline schema started being written, we created a launch interface. This isn't super relevant here so I'll go light on the details, but it basically works in the same way as the builder above - you do nf-core launch user/repo or nf-core launch /path/to/pipeline and it pings you off to a web GUI. If the pipeline has no schema it uses the build functionality to create one without customisation. One difference is that this tool can also run offline using command line wizard prompts (complete with markdown rendering on the terminal 😍 ). Good for people running sensitive stuff or using offline clusters.

Documentation

Once you have this document as simple JSON outside of Nextflow, you can do all sorts. The website parameter documentation is now built with it if found, and command line help is also coming soon. One place to write the docs and it can be used everywhere.

Validation

Ok, the final piece of the puzzle was to use these nice schema validation rules to actually validate some input data. The finishing touches are being put to this for the pipeline template (which will by synced out to all @nf-core pipelines) now in nf-core/tools#852

@KevinMenden did an awesome job working with this to make it work with a groovy library to handle the complex JSON Schema validation, and also bundling this into a zip file in the template so that the pipelines will work offline without needing to download any dependencies. This ability to work without an internet connection is super important. This is my main concern with a plugin approach as described above.

Whilst we were there, we also added a check for parameters with the same name as any core Nextflow option (that will now become an @nf-core guideline, that these must be avoided). And we have some warnings about unexpected params, though these don't halt execution by themselves as they could be quite common.

Conclusion

Ok, essay done. We would love for this all to become a Nextflow standard instead of just nf-core, that was always the plan. We will roll out our own approach above as it's basically done already, but if core Nextflow functionality can be added for validation that would be amazing and we can remove that code again.

[ewels]

A couple of things that I was reminded about after a tweet by @aunderwo - just like nested parameters, JSON schema can support:

• if / then / else logic (see docs)
• Parameter combination using allOf, anyOf, oneOf, not (see docs)

Just like the nested objects mentioned above (params.foo.bar) these will all be ignored by the @nf-core builder and launcher tools. However, because we use standard libraries for validating inputs against the schema, they should work against the entire JSON Schema spec. So if users write schema by hand then in theory the workflow should be able to validate inputs properly using this more complex logic.

All three of these were considered during design but moved to the "maybe one day" pile. Hopefully the nf-core tools will only ignore them and not break if encountered, but of course I don't really know until we try 😅

[pditommaso]

Wow, we have enough material for a paper! 😅

Now, I think the goal of this discussion is to implement a common interface for the schema the json schema validation so that it can be used by any NF pipeline other than then nf-core pipelines and, above all, not to replace all the amazing nf-core tool features ie. builder, launcher and docs.

My understanding is that the only technical problem is the following:

This ability to work without an internet connection is super important. This is my main concern with a plugin approach as described above.

This is a very good point. Currently, the plugins mechanism pull the plugin archive (a zip) from the internet when it cannot find it locally. Currently, it looks in the $HOME/.nextflow/plugins directory but it could be easily extended to search the plugins also in other local paths.

It could be enough to extend it to allow the search of plugins also in the pipeline directory PROJECT/plugins (?). That's basically what you are doing already, but managing the validation JARs in libs/ directory. Does it make sense ?

[pditommaso]

Nice. it should be straightforward to integrate. Do you have the list of dependencies?

[ewels]

Note that we also have a library for generating command-line help from the schema, which we could presumably do something similar with.. That's currently in the dsl2 template branch here and here.

[KevinMenden]

Don't have that list anymore but I'll just go ahead and get it 👍

[KevinMenden]

But basically it's just https://github.com/everit-org/json-schema and org.json, and then the jar file we have contains all their dependencies as well. But that should probably be automatically resolved, right?
Otherwise I can get the full list

[pditommaso]

But that should probably be automatically resolved, right

Indeed

Note that we also have a library for generating command-line help from the schema

let's start with validation, then we can see if makes sense to include other features.

[mes5k]

👍 Excited for this feature!

[ewels]

It's taken two and a half years, but we're finally getting close 😆

[pditommaso]

Ok, I've created the project skeleton https://github.com/nextflow-io/nf-validation. The main code is here

https://github.com/nextflow-io/nf-validation/blob/master/src/main/nextflow/validation/NfcoreSchema.groovy

Still not doing anything in concrete, but it should not be much hard. Let's move the discussion in that project.

[pditommaso]

If you prefer to keep in nf-core org, no problem. I can transfer the project there if you prefer.

[ewels]

This is great! If the idea is to make this a global Nextflow thing and no longer just nf-core, then I think having it in the main Nextflow org is best 👍🏻

[junjun-zhang]

It's great to see this moving forward to getting Nextflow native support. Kudos to the nf-core team (@ewels et al) for the excellent work!

@pditommaso just my two cents: would nf-params-validation or params-validation be slightly better name for the plug-in? Just feel that nf-validation might mislead people to think it's for Nextflow code validation.