The pipestat schema should be a JSON Schema.

[donaldcampbelljr]

The pipestat schema should be a JSON Schema. This will make it more familiar to people.

[donaldcampbelljr]

Some notes on this:

1. Right now, pipestat takes a config_dict or config file and validates it using a json schema validator:
   

   pipestat/pipestat/pipestat.py

   Lines 94 to 96 in 7de7b5e

   	self[CONFIG_KEY] = YAMLConfigManager(entries=config_dict, filepath=self._config_path)
   	_, cfg_schema = read_yaml_data(CFG_SCHEMA, "config schema")
   	validate(self[CONFIG_KEY].exp, cfg_schema)

However, it appears that we currently do not perform json schema validation on the output_schema. This make sense since it is not a JSON Schema. Should we perform this validation during schema processing (in parsed_schema.py)?

2. I assume we will still use a .yaml for the output schema, it will simply need to conform to JSON schema requirements.

3. From some reading, there are some subtle differences between the current .yaml schemas and a JSON schema in yaml format. For example:

Current example of an output schema as a .yaml:

pipeline_name: default_pipeline_name  
samples:  
  collection_of_images:  
    description: "This store collection of values or objects"  
    type: array  
    items:  
      properties:  
          prop1:  
            description: "This is an example file"  
            type: file  
  output_file_in_object:  
    type: object  
    properties:  
      prop1:  
        description: "This is an example file"  
        type: file  
      prop2:  
        description: "This is an example image"  
        type: image  
    description: "Object output"  
  output_file:  
    type: file  
    description: "This a path to the output file"  
  output_image:  
    type: image  
    description: "This a path to the output image"

The same schema in proper JSON schema format (as a .yaml):

$schema: https://json-schema.org/draft/2020-12/schema
$id: https://example.com/output_schema.json
type: object
title: Default Pipeline
description: A pipeline that collects images and outputs a file and an image.
properties:
  pipeline_name:
    type: string
    default: default_pipeline_name
  samples:
    type: object
    properties:
      collection_of_images:
        type: array
        description: A collection of images.
        items:
          type: object
          properties:
            prop1:
              type: file
              description: An example file.
      output_file_in_object:
        type: object
        description: An object containing output files.
        properties:
          prop1:
            type: file
            description: An example file.
          prop2:
            type: image
            description: An example image.
  output_file:
    type: file
    description: The path to the output file.
  output_image:
    type: image
    description: The path to the output image.

My understanding regarding $id and $schema:
-if $schema is not declared the latest draft will be used (which should we use?)
-if $id is not declared, the schema can only be used locally (this seems fine).

Next steps:

1. Change schemas to conform to JSON schema
2. Update schema parsing functions (including recursive functions for complex objects) to handle JSON schemas.
3. Add validation to confirm given output_schema is a valid JSON schema.

Do you have any other thoughts on this @nsheff ?

[nsheff]

I think I'm on the same page as you.

can we use the same thing we already defined for an eido schema?

https://eido.databio.org/en/latest/writing-a-schema/

[donaldcampbelljr]

Regarding the imports section of the eido schema, I believe you can achieve something similar with $ref:
https://json-schema.org/understanding-json-schema/structuring.html#ref

Are you also asking if we should just make the pipestat output_schema an Eido schema? Specifically, to take advantage of checking for files' existence which is outside the scope of JSON schema validation? Do we even need (or desire) to do that for a pipestat output_schema?

[donaldcampbelljr]

Basically we do want to be able to use a JSON schema that is extendable, similar to Eido.

Example:
Eido validates sample tables and has an expected input schema (which are peps).
It uses JSON schema but extends it, e.g. imports and required files. During JSON validation, it ignores items that it does not recognize such as imports. Eido knows what to do with these items and processes them appropriately after JSON validation.

We want to have pipestat use JSON schema that is extendable.
To consider:

• Can we replace imports with $ref?
• Make project attributes parallel to samples. Eido currently only validates samples.
• Make pipeline_name optional?

[donaldcampbelljr]

I left pipeline_name as being required via the output schema.
Pipestat can now accept output schemas that are JSON schema OR continue using the original output schema for backwards compatibility.

[nsheff]

I noticed you used project as the attribute name for the project-level info. In eido, we used config as that name. So, these will not be compatible.

I think we should use the same standard. But, I actually like project better than config. So... what do you think?

[donaldcampbelljr]

In the context of pipestat, project makes more sense to me than having it be under config. This is because we also have a pipestat config file which is separate (this could become confusing to the user).

[donaldcampbelljr]

Per discussion, we would like eido to use the same attribute names as pipestat (project vs config) while still maintaining backwards compatibility.

[nsheff]

Can you provide an example of a true JSON schema pipestat schema? I think the above examples are all not quite right... they lack the project section

[nsheff]

Here's an idea for that, now using $defs and $ref instead of our custom replacements.

$schema: https://json-schema.org/draft/2020-12/schema
$id: https://example.com/output_schema.json
$defs:
  image:
    type: object
    object_type: image
    properties:
      path:
        type: string
      thumbnail_path:
        type: string
      title:
        type: string
    required:
      - path
      - thumbnail_path
      - title
  file:
    type: object
    object_type: file
    properties:
      path:
        type: string
      title:
        type: string
    required:
      - path
      - title
type: object
title: Default Pipeline
description: A pipeline that collects images and outputs a file and an image.
properties:
  pipeline_name:
    type: string
    default: default_pipeline_name
  samples:
    type: object
    properties:
      right_file:
        $ref: "#/$defs/file"
        label: bigBed
        description: bigBed file
      collection_of_images:
        type: array
        description: A collection of images.
        items:
          type: object
          properties:
            prop1:
              type: file
              description: An example file.
      output_file_in_object:
        type: object
        description: An object containing output files.
        properties:
          prop1:
            type: file
            description: An example file.
          prop2:
            type: image
            description: An example image.
  project:
    type: object
    properties: 
      output_file:
        type: file
        description: The path to the output file.
      output_image:
        type: image
        description: The path to the output image.

[donaldcampbelljr]

Implemented the above suggestion using $ref and $defs: b71d387

[nsheff]

excellent! Can you try creating new versions of the bbconf schemas following this approach, and see if they will work with bbconf/bedhost?

[donaldcampbelljr]

Yes, we will need to merge these changes to dev via PR #109 first

[donaldcampbelljr]

Per Tuesday's discussion, I attempted to use the built in referencing library to resolve internal references within a JSON schema written in yaml using $ref and $def.

This is not working; I cannot get a resolved version of the schema.

Reading material:
https://python-jsonschema.readthedocs.io/en/stable/referencing/#resolving-references-to-schemas-written-in-yaml
https://readthedocs.org/projects/python-jsonschema/downloads/pdf/latest/

MCVE:

Given a simple schema:

$schema": "https://json-schema.org/draft/2020-12/schema"
$id: "#/$defs/"
title: An example Pipestat output schema
description: A pipeline that uses pipestat to report sample and project level results.
type: object
properties:
  pipeline_name: "default_pipeline_name"
  samples:
    type: object
    properties:
        output_image:
          $ref: "#/$defs/image"
          description: "This an output image"
$defs:
  image:
    type: object
    object_type: image
    properties:
      path:
        type: string
      thumbnail_path:
        type: string
      title:
        type: string
    required:
      - path
      - thumbnail_path
      - title

main.py:

from pathlib import Path
import yaml
from referencing import Registry, Resource
from jsonschema import Draft202012Validator
path_yaml = Path("/home/drc/GITHUB/pythonpractice/jsonschema_practice/sample.yaml") 
contents = yaml.safe_load(path_yaml.read_text())

resource = Resource.from_contents(contents, {"$schema": "https://json-schema.org/draft/2020-12/schema"})

registry = Registry()

registry = registry.with_resources([("#/$defs", resource)])

for ref in registry._uncrawled:
    print(ref)

validator = Draft202012Validator(schema=registry)

validator.validate(registry)

The above code loads the yaml file and the validator has no issues. However, the registry continues to have registry._uncrawled and the validator object does not seem to resolve them.

[nsheff]

Yeah, I can confirm. I've been poking around here for a few hours tonight and also cannot figure out how to get a resolved schema. You can type resource.crawl() but it doesn't help.

The schema that I get out is called a Resolved object, but it still includes $ref items. validation works fine with these. So, I guess we may have to roll our own after all :(

[nsheff]

python-jsonschema/jsonschema#1188 (comment)

[donaldcampbelljr]

Ok, I will merge PR 109 to dev and proceed with implementing these changes in bbconf.

I will keep this issue open to circle back around for:

• implement fetching from https://schema.databio.org/

[nsheff]

Just a few thoughts on this I had this morning:

• the reason we needed the resolving is so that for things like an image, we make the correct sqlmodel object
• this happens right now with a 'type lookup', here:

  pipestat/pipestat/backends/db_backend/db_parsed_schema.py

  Line 321 in 6985fcd

  curr_type_spec = CANONICAL_TYPES[curr_type_name]

  -- if the type is image, then it gets replace by a built-in image model.

we could solve this easily but just having it do this instead:

• when determining what the type of the element is in that function, we first check if the particular element has a $ref propery.
• if there's no $ref property, we just proceed as normal, no change.
• if there is a $ref property, we follow that ref, and then use .update to update the dict describing the type.
• if the $ref points starts with #/$defs, then we look it up in the defs entry of the current.
• if it starts with http, then we retrieve it from URL.
• otherwise, we throw an error that we can't follow the $ref.

The function below (not tested pseudocode) should be all we need:

def resolve_ref(subschema: dict, schema: dict) -> dict:
    """
    Given a subschema, and its schema, resolve the $ref entry (if any)
    and return an equivalent subschema with the $ref resolved

    @param subschema represents the schema for a specific property
    @param schema the parent schema, which may include the $defs for lookup
    """
    if "$ref" in subschema:
        uri = subschema["$ref"]
        if uri.startswith("#/$defs"):
            print("Found ref to local $def:", uri)
            relative_uri = uri.removeprefix("#/$defs")
            resolved_ref = schema["$defs"][relative_uri]  # resolve ref
            subschema.update(resolved_ref)  # update schema
        else if uri.startswith("http"):
            print("Found ref to remote schema: ", uri)
            response = httpx.get(uri)  # resolve ref
            subschema.update(response.json())  # update schema
        else:
            raise NotImplementedError(f"Can't resolve ref: {uri}")
        del subschema["$ref"]  # remove resolved ref
    return subschema