Release v1.3.0.4

This is a significant release with a breaking change, multiple new features and a number of minor fixes and improvements.

rectify replaces update

The subcommand update was renamed to rectify to better express its purpose. The name update suggested that this command could effectively migrate packages from one Poseidon version to the next, which was never the case. Structural and semantical changes to the package always have to be performed by the user through other means, so usually by manually editing the respective package files. rectify only helps to adjust a number of parameters (mostly in the POSEIDON.yml file) after the changes have been applied: It updates checksums, iterates version numbers, adds contributors and appends logging information to CHANGELOG files. Despite this limitation it is still a valuable tool, especially for the management of large package archives, where structural changes are often applied to many packages at once, all requiring "rectification" in the end.

rectify doesn't only introduce a different name, it also features a different interface. While update was a catch-all procedure with an opinionated, default behaviour that could partially be adjusted with various flags, rectify follows a much more transparent opt-in philosophy. The new interface allows to precisely choose which aspects of a package should be updated.

⚠️ Please note that certain changes like incrementing the version number or adding a logText therefore do not happen automatically any more in rectify. They have to be requested explicitly!

Here is the new command line documentation of rectify:

Usage: trident rectify (-d|--baseDir DIR) [--ignorePoseidonVersion]
                       [--poseidonVersion ?.?.?]
                       [--packageVersion VPART [--logText STRING]]
                       [--checksumAll | [--checksumGeno] [--checksumJanno]
                         [--checksumSSF] [--checksumBib]]
                       [--newContributors DSL]

  Adjust POSEIDON.yml files automatically to package changes

Available options:
  -h,--help                Show this help text
  -d,--baseDir DIR         A base directory to search for Poseidon packages.
  --ignorePoseidonVersion  Read packages even if their poseidonVersion is not
                           compatible with trident.
  --poseidonVersion ?.?.?  Poseidon version the packages should be updated to:
                           e.g. "2.5.3".
  --packageVersion VPART   Part of the package version number in the
                           POSEIDON.yml file that should be updated: Major,
                           Minor or Patch (see https://semver.org).
  --logText STRING         Log text for this version in the CHANGELOG file.
  --checksumAll            Update all checksums.
  --checksumGeno           Update genotype data checksums.
  --checksumJanno          Update .janno file checksum.
  --checksumSSF            Update .ssf file checksum
  --checksumBib            Update .bib file checksum.
  --newContributors DSL    Contributors to add to the POSEIDON.yml file in the
                           form "[Firstname Lastname](Email address);...".

serve now provides different package archives and list and fetch can access them

trident serve, so the subcommand behind the server providing the Poseidon Web API, can now host packages from multiple named archives in parallel. This works through a modified -d interface on the command line and a new option ?archive=... in the Web API. The client commands fetch and list can request information and package download from these different archives with a new option --archive. If --archive (or ?archive=... in the http request) are not given, then the server falls back to a default archive (the first in -d).

See the Poseidon public archive and Web API documentation for the concrete consequences of this new feature.

validate can now check individual files

The validate subcommand is no longer confined to validating entire poseidon packages. It can still very much do so with -d, where -- just as before -- a number of optional flags can be used to control the exact behaviour. This release, in fact, adds the new options --ignorePoseidonVersion and --ignoreChecksums here. But validate can now also read, parse and thus check individual files: POSEIDON.yml files, genotype data files, .janno files, .ssf files or .bib files. This is tremendously useful for building packages step-by-step, e.g. in automatic pipelines.

⚠️ Please note that these individual file checks naturally do not include the cross-file checks (e.g. publication keys across .janno and .bib). These are only available in the full package validation process.

Here is the new command line documentation of validate:

Usage: trident validate ((-d|--baseDir DIR) [--ignoreGeno] [--fullGeno]
                          [--ignoreDuplicates] [-c|--ignoreChecksums]
                          [--ignorePoseidonVersion] |
                          --pyml FILE | (-p|--genoOne FILE) | --inFormat FORMAT
                          --genoFile FILE --snpFile FILE --indFile FILE |
                          --janno FILE | --ssf FILE | --bib FILE) [--noExitCode]

  Check Poseidon packages or package components for structural correctness

Available options:
  -h,--help                Show this help text
  -d,--baseDir DIR         A base directory to search for Poseidon packages.
  --ignoreGeno             Ignore snp and geno file.
  --fullGeno               Test parsing of all SNPs (by default only the first
                           100 SNPs are probed).
  --ignoreDuplicates       Do not stop on duplicated individual names in the
                           package collection.
  -c,--ignoreChecksums     Whether to ignore checksums. Useful for speedup in
                           debugging.
  --ignorePoseidonVersion  Read packages even if their poseidonVersion is not
                           compatible with trident.
  --pyml FILE              Path to a POSEIDON.yml file.
  -p,--genoOne FILE        One of the input genotype data files. Expects .bed,
                           .bim or .fam for PLINK and .geno, .snp or .ind for
                           EIGENSTRAT. The other files must be in the same
                           directory and must have the same base name.
  --inFormat FORMAT        The format of the input genotype data: EIGENSTRAT or
                           PLINK. Only necessary for data input with --genoFile
                           + --snpFile + --indFile.
  --genoFile FILE          Path to the input geno file.
  --snpFile FILE           Path to the input snp file.
  --indFile FILE           Path to the input ind file.
  --janno FILE             Path to a .janno file.
  --ssf FILE               Path to a .ssf file.
  --bib FILE               Path to a .bib file.
  --noExitCode             Do not produce an explicit exit code.

Other, minor changes

• Fixed the behaviour of forge when combining .bib files. Publication duplicates are now properly removed upon merging and the output is alphabetically sorted.
• Added a global option --debug, which is short for --logMode VerboseLog.
• Refactored summarise to make the resulting counts more accurate. Some variables in the output table have been renamed as well.
• Fixed the behaviour of chronicle when updating a chronicle file (with -u): The lastModified field is now only touched if there is actually a change in the package list.
• Some cleaning of the general trident command line documentation: Added meaningful meta variables to all arguments.
• Shortened the default command line output of fetch to make it more readable.
• Slightly better error handling for failed http requests in fetch and list --remote.