Merge pull request #27 from pepkit/dev

v0.1.5

.github/workflows/run-pytest.yml

@@ -32,7 +32,7 @@ jobs:
       run: python -m pip install .
 
     - name: Run pytest tests
-      run: pytest tests --cov=./ --cov-report=xml
+      run: pytest tests -x -vv --cov=./ --cov-report=xml
 
     - name: Upload coverage to Codecov
       uses: codecov/codecov-action@v1

.pre-commit-config.yaml

@@ -0,0 +1,20 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v3.4.0
+    hooks:
+      - id: trailing-whitespace
+      - id: check-yaml
+      - id: end-of-file-fixer
+      - id: requirements-txt-fixer
+      - id: trailing-whitespace
+
+  - repo: https://github.com/PyCQA/isort
+    rev: 5.7.0
+    hooks:
+      - id: isort
+        args: ["--profile", "black"]
+
+  - repo: https://github.com/psf/black
+    rev: 20.8b1
+    hooks:
+      - id: black

docs/README.md

@@ -6,7 +6,7 @@
 
 ## Introduction
 
-Eido is a validation tool for [PEPs](http://pepkit.github.io) based on [JSON Schema](https://github.com/Julian/jsonschema). The PEP specification defines a formal structure for organizing project and sample metadata, and eido provides a way to validate if data complies with that specification. Eido extends the JSON Schema vocabulary with PEP-specific features, like required input files.
+Eido is a validation and format conversion tool for [PEPs](http://pepkit.github.io). It provides validation based on [JSON Schema](https://github.com/Julian/jsonschema). The PEP specification defines a formal structure for organizing project and sample metadata, and eido provides a way to validate if data complies with that specification. Eido extends the JSON Schema vocabulary with PEP-specific features, like required input files. Eido also provides a command-line interface to convert a PEP input into a variety of outputs using [eido filters](filters.md), and includes ability to write [custom filters](writing-a-filter.md).
 
 ## Why do we need eido?
 
@@ -34,6 +34,6 @@ An eido schema is written using the JSON Schema vocabulary, plus a few additiona
 
 ## What does 'eido' mean?
 
-*Eidos* is a Greek term meaning *form*, *essence*, or *type* (see Plato's [Theory of Forms](https://en.wikipedia.org/wiki/Theory_of_forms)). Schemas are analogous to *forms*, and eido tests claims that an instance is of a particular form.
+*Eidos* is a Greek term meaning *form*, *essence*, or *type* (see Plato's [Theory of Forms](https://en.wikipedia.org/wiki/Theory_of_forms)). Schemas are analogous to *forms*, and eido tests claims that an instance is of a particular form. Eido also helps *change* forms using filters.
 
 

docs/api_docs.md

@@ -0,0 +1,151 @@
+Final targets: convert_project, get_available_pep_filters, inspect_project, read_schema, validate_config, validate_project, validate_sample
+<script>
+document.addEventListener('DOMContentLoaded', (event) => {
+  document.querySelectorAll('h3 code').forEach((block) => {
+    hljs.highlightBlock(block);
+  });
+});
+</script>
+
+<style>
+h3 .content {
+    padding-left: 22px;
+    text-indent: -15px;
+ }
+h3 .hljs .content {
+    padding-left: 20px;
+    margin-left: 0px;
+    text-indent: -15px;
+    martin-bottom: 0px;
+}
+h4 .content, table .content, p .content, li .content { margin-left: 30px; }
+h4 .content {
+    font-style: italic;
+    font-size: 1em;
+    margin-bottom: 0px;
+}
+
+</style>
+
+
+# Package `eido` Documentation
+
+
+Project configuration
+
+```python
+def validate_project(project, schema, exclude_case=False)
+```
+
+Validate a project object against a schema
+#### Parameters:
+
+- `project` (`peppy.Sample`):  a project object to validate
+- `schema` (`str | dict`):  schema dict to validate against or a path to one
+- `exclude_case` (`bool`):  whether to exclude validated objectsfrom the error. Useful when used ith large projects
+
+
+
+
+```python
+def validate_sample(project, sample_name, schema, exclude_case=False)
+```
+
+Validate the selected sample object against a schema
+#### Parameters:
+
+- `project` (`peppy.Project`):  a project object to validate
+- `sample_name` (`str | int`):  name or index of the sample to validate
+- `schema` (`str | dict`):  schema dict to validate against or a path to one
+- `exclude_case` (`bool`):  whether to exclude validated objectsfrom the error. Useful when used ith large projects
+
+
+
+
+```python
+def validate_config(project, schema, exclude_case=False)
+```
+
+Validate the config part of the Project object against a schema
+#### Parameters:
+
+- `project` (`peppy.Project`):  a project object to validate
+- `schema` (`str | dict`):  schema dict to validate against or a path to one
+- `exclude_case` (`bool`):  whether to exclude validated objectsfrom the error. Useful when used ith large projects
+
+
+
+
+```python
+def read_schema(schema)
+```
+
+Safely read schema from YAML-formatted file.
+
+If the schema imports any other schemas, they will be read recursively.
+#### Parameters:
+
+- `schema` (`str | Mapping`):  path to the schema fileor schema in a dict form
+
+
+#### Returns:
+
+- `list[dict]`:  read schemas
+
+
+#### Raises:
+
+- `TypeError`:  if the schema arg is neither a Mapping nor a file path orif the 'imports' sections in any of the schemas is not a list
+
+
+
+
+```python
+def inspect_project(p, sample_names=None, max_attr=10)
+```
+
+Print inspection info: Project or, if sample_names argument is provided, matched samples
+#### Parameters:
+
+- `p` (`peppy.Project`):  project to inspect
+- `sample_names` (`Iterable[str]`):  list of samples to inspect
+- `max_attr` (`int`):  max number of sample attributes to display
+
+
+
+
+```python
+def get_available_pep_filters()
+```
+
+Get a list of available target formats
+#### Returns:
+
+- `List[str]`:  a list of available formats
+
+
+
+
+```python
+def convert_project(prj, target_format, plugin_kwargs=None)
+```
+
+Convert a `peppy.Project` object to a selected format
+#### Parameters:
+
+- `prj` (`peppy.Project`):  a Project object to convert
+- `plugin_kwargs` (`dict`):  kwargs to pass to the plugin function
+- `target_format` (`str`):  the format to convert the Project object to
+
+
+#### Raises:
+
+- `EidoFilterError`:  if the requested filter is not defined
+
+
+
+
+
+
+
+*Version Information: `eido` v0.1.5-dev, generated by `lucidoc` v0.4.2*

docs/autodoc_build/eido.md

@@ -7,7 +7,7 @@ document.addEventListener('DOMContentLoaded', (event) => {
 </script>
 
 <style>
-h3 .content { 
+h3 .content {
     padding-left: 22px;
     text-indent: -15px;
  }
@@ -18,7 +18,7 @@ h3 .hljs .content {
     martin-bottom: 0px;
 }
 h4 .content, table .content, p .content, li .content { margin-left: 30px; }
-h4 .content { 
+h4 .content {
     font-style: italic;
     font-size: 1em;
     margin-bottom: 0px;
@@ -110,7 +110,95 @@ Print inspection info: Project or, if sample_names argument is provided, matched
 
 
 
+```python
+def get_available_pep_filters()
+```
+
+Get a list of available target formats
+#### Returns:
+
+- `List[str]`:  a list of available formats
+
+
+
+
+```python
+def convert_project(prj, target_format, plugin_kwargs=None)
+```
+
+Convert a `peppy.Project` object to a selected format
+#### Parameters:
+
+- `prj` (`peppy.Project`):  a Project object to convert
+- `plugin_kwargs` (`dict`):  kwargs to pass to the plugin function
+- `target_format` (`str`):  the format to convert the Project object to
+
+
+#### Raises:
+
+- `EidoFilterError`:  if the requested filter is not defined
+
+
+
+
+```python
+def basic_pep_filter(p, **kwargs)
+```
+
+Basic PEP filter, that does not convert the Project object.
+
+This filter can save the PEP representation to file, if kwargs include `path`.
+#### Parameters:
+
+- `p` (`peppy.Project`):  a Project to run filter on
+
+
+
+
+```python
+def yaml_pep_filter(p, **kwargs)
+```
+
+YAML PEP filter, that returns Project object representation.
+
+This filter can save the YAML to file, if kwargs include `path`.
+#### Parameters:
+
+- `p` (`peppy.Project`):  a Project to run filter on
+
+
+
+
+```python
+def csv_pep_filter(p, **kwargs)
+```
+
+CSV PEP filter, that returns Sample object representations
+
+This filter can save the CSVs to files, if kwargs include
+`sample_table_path` and/or `subsample_table_path`.
+#### Parameters:
+
+- `p` (`peppy.Project`):  a Project to run filter on
+
+
+
+
+```python
+def yaml_samples_pep_filter(p, **kwargs)
+```
+
+YAML samples PEP filter, that returns only Sample object representations.
+
+This filter can save the YAML to file, if kwargs include `path`.
+#### Parameters:
+
+- `p` (`peppy.Project`):  a Project to run filter on
+
+
+
+
 
 
 
-*Version Information: `eido` v0.1.4, generated by `lucidoc` v0.4.3*
+*Version Information: `eido` v0.1.5-dev, generated by `lucidoc` v0.4.2*

docs/changelog.md

@@ -1,6 +1,22 @@
 # Changelog
 
-This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format. 
+This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format.
+
+## [0.1.5] - 2021-04-15
+### Added
+
+** The PEP filters are an experimental feature and may change in feature versions of `eido`**
+
+- `eido convert` command that converts the provided PEP to a specified format
+- `eido filter` command that lists available filters in current environment
+- built-in plugins:
+  - `basic_pep_filter`
+  - `yaml_pep_filter`
+  - `csv_pep_filter`
+  - `yaml_samples_pep_filter`
+
+### Changed
+- in `validate_inputs` function sample attributes are first validated for existence before their values' existence is checked
 
 ## [0.1.4] - 2021-03-14
 ### Changed
@@ -35,13 +51,13 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 
 ### Changed
 - previous CLI `eido` functionality moved to `eido validate`
- 
+
 ## [0.0.6] - 2020-02-07
 ### Changed
 - CLI can accommodate URLs.
 
 ## [0.0.5] - 2020-02-04
-### Added 
+### Added
 - [documentation website](http://eido.databio.org/en/latest/)
 - include version in the CLI help
 
@@ -50,7 +66,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - `validate_sample` function for sample level validation
 - sample validation CLI support (via `-n`/`--sample-name` argument)
 - `validate_config` to facilitate samples exclusion in validation
-- config validation CLI support (via `-c`/`--just-config` argument)  
+- config validation CLI support (via `-c`/`--just-config` argument)
 
 ## [0.0.3] - 2020-01-30
 ### Added