Rewrite for sc (#24)

* Treat files properly

* Move python scripts to dedicated directory

* Move jinja templates to dedicated directory

* Remove old script

* Modularize relative import in python script

* Update CMake and run python as module

* Factor out file opening

* Tidy up old comments and dead code

* Factor out hardcoded default doc data

* Move lower-casing into templates, in prep for SC 

Temporarily comment out fft sub params that were being missed before, so 
that interim tests passed

* Fix RST errors

* Add draft docutils writer for rst 2 scdoc

* Some tidying and commenting, ahead of substantial refactor

* Refactor for modularity

* Improe logging and reporting

* Add some doc and tidy a bit

* move scdoc docutils writer

* rename scdoc writer

* Add scdoc generation

still todo: 
* weird cases with multiple inputs like NMFMorph and AudioTransport
* cleverer categorisation

* Fix control lookup and eliminate some duplication in validation code

* Enable per-host defaults and fix lookup problems. 

Also prettify log messages

* Add requirements.txt

* tidy drivers and defaults a bit

* add SC example code files 

harvested from *current* help files (script also committed)

* include sc example code from external files in rendered template

* minor tidying to SC child templates

* write warning log to file and add option to surpress console output

* housekeeping

* enable testing with CTest

* fix missing arg 'type' in merge, add test

* test merging, fix validation tests

* Update README

* Add some very rough code documentation

* update gitignore

* Robustly handle case where message's `args` block is completely missing

* correct templates folder

* fix whacky indentation

* scdoc template: setters don't need explicit handling

* Don't die if a driver passes defaults as None

* Correct type for data in CLI namer

* get rid of double dots in topic file extentions

* Stop SC being sad if RELATED is empty

* rst to scdoc: render section headings and properly separate paragraphs

* Don't generate rst in undocumented argument names

* Properly escape plural in mlprgressor doc

.gitignore

@@ -8,3 +8,5 @@ sphinxout
 doctrees
 schelp
 *.pyc
+interfaces/*
+*.log

CMakeLists.txt

@@ -26,7 +26,8 @@ set(FLUID_PATH "" CACHE PATH "The top level of the flucoma-core repo. Will clone
 set(MAX_DOC_OUT "${CMAKE_BINARY_DIR}/max_ref" CACHE PATH "")
 set(PD_DOC_OUT  "${CMAKE_BINARY_DIR}/pd_ref" CACHE PATH "")
 set(CLI_DOC_OUT  "${CMAKE_BINARY_DIR}/cli_ref" CACHE PATH "") 
-
+set(SC_DOC_OUT  "${CMAKE_BINARY_DIR}/sc_ref" CACHE PATH "") 
+# option(TESTS,"Run tests?")
 include(FetchContent)
 
 set(FETCHCONTENT_QUIET FALSE)
@@ -82,7 +83,7 @@ endif()
 
 set(FLUID_JSON_DOC "${CMAKE_BINARY_DIR}/json")
 set(FLUID_YAML_DOC "${CMAKE_CURRENT_SOURCE_DIR}/doc")
-set(FLIUD_JINJA_TEMPLATES "${CMAKE_CURRENT_SOURCE_DIR}/script/templates")
+set(FLIUD_JINJA_TEMPLATES "${CMAKE_CURRENT_SOURCE_DIR}/flucoma/doc/templates")
 
 # Make sure to build targets from flucoma-core
 macro(subdirlist result curdir)
@@ -125,14 +126,36 @@ macro(add_ref_target platform comment)
 
   add_custom_target(MAKE_${platform_uc}_REF
     COMMAND ${CMAKE_COMMAND} -E make_directory "${${platform_uc}_DOC_OUT}"
-  	COMMAND ${PYTHON_3} "${CMAKE_CURRENT_SOURCE_DIR}/script/MakeRef.py" "${platform_lc}" ${FLUID_JSON_DOC} ${FLUID_YAML_DOC} "${${platform_uc}_DOC_OUT}" ${FLIUD_JINJA_TEMPLATES}
+  	COMMAND ${PYTHON_3} "-mflucoma.MakeRef" "${platform_lc}" ${FLUID_JSON_DOC} ${FLUID_YAML_DOC} "${${platform_uc}_DOC_OUT}" ${FLIUD_JINJA_TEMPLATES}
+    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
   	COMMENT ${comment}
     DEPENDS ${json_files}
   	VERBATIM
   )
 endmacro()
 
-
 add_ref_target(max "Making Max ref")
 add_ref_target(pd "Making PD ref")
 add_ref_target(cli "Making CLI ref")
+add_ref_target(sc "Making SC ref")
+
+enable_testing()
+
+execute_process(
+  COMMAND "python" "list_python_tests.py" 
+  OUTPUT_VARIABLE FLUCOMA-DOC-PYTHON_TESTS
+  WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+  OUTPUT_STRIP_TRAILING_WHITESPACE
+  ERROR_STRIP_TRAILING_WHITESPACE
+)
+
+separate_arguments(TEST_LIST UNIX_COMMAND ${FLUCOMA-DOC-PYTHON_TESTS})
+
+foreach(THIS_TEST ${TEST_LIST})
+    message(STATUS ${THIS_TEST})
+    add_test(
+      NAME ${THIS_TEST} 
+      COMMAND ${PYTHON_3} "-m" "unittest" "-v" ${THIS_TEST}
+      WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+    )
+endforeach()

README.md

@@ -1,20 +1,40 @@
-Generate JSON files of parameters from [Fluid Corpus Manipulation Library](https://github.com/flucoma/flucoma-core) objects
+Produce documentation for [Fluid Corpus Manipulation Library](https://github.com/flucoma/flucoma-core) objects, from a combination of handwritten material and generated data. 
 
-This is quick and dirty, so don't be surprised at weirdness.
+## Overview 
+The generated data comes from building an running a set of C++ executables that produce JSON files detailing the interface of FluCoMa objects (parameter and message names, etc.). 
 
-The JSON will turn up in a folder called `json` in the top level directory, and is generated as part of compilation (by CMake).
+The handwritten documentation is (currently) in YAML files, that are meant to match up with the generated data. Some Python code then makes sure this assumption is true (i.e. validates the completeness of the handwritten docs against the generated data), and produces rendered documentation for a choice of environments (Max, Pure Data, Supercollider and the command line). 
 
-So, simple build steps should be along the lines of
+This is intended principally as internal tooling, so there may well be rough edges and weird things. Operation on platforms other than Mac OS or Linux is untested. 
+
+## Quick Start 
+
+The routine way to interact with everything is via CMake. In addition to this you will need: git, a working C++ toolchain, and Python >= 3.8 with the packages listed in `requirements.txt`. (use `pip install -r requirements.txt` to install all these in one go. )
+
+To configure the project and generate the JSON run
 ```bash
 mkdir build
 cd build
-cmake -DFLUID_PATH=/your/path/to/your/fluid-corpus-manipulation/repo ..
+cmake ..
 make
 ```
 
-This requires python 3 with the docutils, jinja2, and pyYAML (>=5.1)  packages. 
+If you have a local checkout of the `flucoma-core` repo, then this can be used by adding `-DFLUID_PATH=<path to your flucoma core>`. 
+
+The JSON files are generated into `build/json`.
+
+The CMake exposes targets for producing docs for particular environments (`MAKE_MAX_REF`, `MAKE_PD_REF`, `MAKE_SC_REF`,`MAKE_CLI_REF`). These targets will (re-)generate the JSON if needed and then run a Python script to produce the actual docs. 
+
+The output paths for the docs default to `build/<host>_ref` but can be overridden by setting CMake cache variables (`MAX_DOC_OUT`,`PD_DOC_OUT`,`CLI_DOC_OUT`,`SC_DOC_OUT`)
+
+### Python Herding
+On platforms with multiple Pythons installed (like Macs), CMake can sometimes need some heavy hints in order to avoid settling on system Python. If you get error messages about missing packages, it is likely that CMake has found the 'wrong' one. 
+
+For recent versions of CMake (>= 3.15?), adding `-DPYTHON_EXECUTABLE=<path to your python of choice>` is a blunt instrument, but normally works. You can find the path via `which python` in the terminal, if needed. Considerably more options can be found in the [documentation for CMake's `find_python` module](https://cmake.org/cmake/help/latest/module/FindPython.html). 
+
+## Documentation 
 
-If you have multiple python 3s on your system, using variables like CMAKE_FIND_FRAMEWORK might help CMake find the right one. 
+Is still thin, but will gather mass in the `code-docs` directory. 
 
 ## Credits 
 #### FluCoMa core development team (in alphabetical order)

code-docs/index.md

@@ -0,0 +1,96 @@
+
+# flucoma-docs code proto-documentation 
+
+Some code proto-documentation, aimed at those who need to extend or help maintain `flucoma-docs`. The very high level sketch is that there's some C++ code that scrapes some data from the `Clients` in `flucoma-core`, and some Python code that unites this data with structured, handwritten documentation and renders the result into a choice of formats, targeting the environments Flucoma deploys to. 
+
+## The C++ Bit 
+
+We shall gloss over for now, because it has Change In its Future, pending some significant refactoring and reorganising work on `flucoma-core` in the coming months. It's unfortunately pretty hard to follow unless you know the details of the innards of core, and no-one but me fits that brief currently. In any case, yes I know it takes a while to build. No, I don't know when that will improve. 
+
+
+## The Python Bit 
+
+The Python portion has recently had quite an extensive reorganisation, to transform it from the well-known idiom of one-hooj-scary-function, and to pave the way for extending it to produce Supercollider docs. This means it's now a good deal more readable–and documentable–than before, albeit still slightly shonky almost everywhere you look. 
+
+Below is the basic flow:
+
+![python code flow](./flow.png)
+
+The Python code all lives in the `flucoma` folder and its children (and so, is in various subpackages of `flucoma` in Python). The main entry point is `flucoma.MakeRef`, which is normally invoked via CMake (but can be invoked by hand). It expects 
+* a string telling it which CCE to render for 
+* a collection of paths telling it where to find data, and where to write output. 
+
+The `main` function parses the program arguments, dynamically loads a 'driver' for the required host (see below), and then goes through the steps of validating, merging and rendering for each file it encounters in the JSON folder.  
+
+Nothing else currently lives in the top level package: all the action is down in `flucoma.doc`.
+
+### Data (`flucoma.doc.data`)
+
+* We have JSON files, containing the generated data for a `Client`. This is taken to be the canonical Source of Truth for the names, number, types, etc. of parameters / controls, messages and so forth exposed by the object (because it comes from the code that defines the object). 
+
+* We also have YAML files, that have actual human crafted words that explain these various data. They are only YAML for now – we've decided to port them to reStructuredText once this update is settled. But YAML they are, which makes them easy for computers, but fiddly for humans. 
+
+* Blocks of prose *within* the YAML, however, still might want some inline markup, for emphasis, lists, links to other objects etc. *That's* in reStructuredText, just to add to the fiddly for humans bit (hence the forthcoming change). 
+
+* Nonetheless, basic takeaway: two sources of data, and they may or may not match. If they *don't* match, it would be good to tell a human person, so that the handwritten documentation can be brought back into sync with all that is good and true. 
+
+### Validation (`flucoma.doc.validate`)
+
+This is where the most complex work happens. Given our canonical generated data for each object, we want to check that the handwritten documentation covers everything, and warn the world if there's stuff missing. However, there are instances where parameters or messages that are common across a bunch of objects might have some common documentation to be used, to cut down on redundancy and chances for inconsistency. There are also occasional slight differences between different environments and how stuff needs to be documented. So, things can get a bit edge-casey. 
+
+Validation proceeds hierarchically. First through the object, checking for the top-level stuff like a digest and description for each Client, and then through controls (parameters) and messages (and message arguments). 
+
+The original incarnation of this code was a forest of `if` trees and manual `dict` mining. It wasn't nice, and was almost impossible to extend, so now we (ab)use the [schema](https://github.com/keleshev/schema) package to try for something a little clearer and more configurable. To be frank, it's still littered with code crimes, but hopefully more legible ones (and definitely more elegant). `Schema` basically works by being fed a data structure the same-ish shape as the thing you want to validate, and checking input against that. Because we're deriving Our Truth from the JSON, we build these structures on the fly. 
+
+`Schema` also allows for neat stuff, like denoting certain things as optional, or giving different options for how a value can be valid. However, its basic inclination when something is missing is to bail, whereas the strategy here is *always* to complete, but to provide meaningful and annoying feedback when stuff is missing. In any case where something is missing, we will fallback to checking to see if there is a 'default' for it. There are two sources of these defaults: one global (`flucoma.docs.defaults`), and one host-specific (provided by the 'driver': each host sub-package has a `defaults.py`). Host-specific entries will always override global ones. If a lookup fails, then a warning is issued to the logging system *and* rendered into the generated documentation. Y'know, for motivation. 
+
+The code to do all this is a bit gnarly, and a future version of these docs will go into much more detail about how its achieved, but the potted version: 
+1. doing this fallback trick with `Schema` at the same time as ascertaining what was there and what was missing seemed too horrible, so the validation for controls ('parameters' in soon-to-be-obsolete-speak) and messages happens in two passes, and missing data is filled out with `None`. 
+2. the `Fallback` stuff itself lives in `flucoma.doc.validate.common`, and takes some mild liberties in order to try and do lookups through hierarchically structured data. 
+3. our desire to actually tell the world useful things about what is missing results in the greatest number of code crimes and gratuitous cuteness. We're using the built-in `logging` package, and would like to be able to issue warnings that record which object, control, message argument or whatever is missing documentation. To do this, we have a 'context' variable, which is updated using Python a `contextmanager` (basically a thing you can use in a `with` block). To be able to actually keep that up to date whilst `Schema` does its stuff has involved making a little `Schema` wrapper (`RecordContext`) in `common` that records the context as it iterates over structures. At the cost of readability. 
+
+### Merging 
+
+Once we've finished checking and fleshing out our documentation, the generated and handwritten data are merged to a single `dict`y structure to be passed on to rendering. At this stage, controls that are non-modulate-able are separated off, because each host treats them differently to modulate-able ones. 
+
+However, because the needs of different environments are, well, different, there is a post-processing stage following the merge. The most glaring difference here is that whilst for Max and PD we need our controls and messages to be sorted, in SC they need to remain in the order declared. 
+
+### Rendering 
+
+Still happens via `Jinja`. The templates for Max, PD and CLI are substantively unchanged. There are new templates for SC, which use Jinja's inheritance feature so that the very different shapes of different species of object can be parcelled off cleanly. 
+
+As with the validation, I've had to do some cute things in order to try and keep track of 'context' whilst the render is going on, in order to log any warnings from docutils when it renders reStructuredText. The gist of this is that a view type is wrapped around the dictionaries for controls and messages that records the current context when they are iterated by Jinja (see `flucoma.doc.logger.ContextView`). 
+
+A note on rendering links to other Flucoma objects. The most frequent cause of reStructuredText errors seems to be with trying to pluralise a `:fluid-obj:` reference like 
+
+```reStructuredText
+.. won't work:
+:fluid-object:`DataSet`s  
+``` 
+I don't have a completely good solution to this yet. reStructuredText reference syntax would let us have different text, whilst still linking to the correct entity:
+
+```reStructuredText
+:fluid-obj:`DataSets<DataSet>`
+```
+
+would print `DataSets` but still link properly in SC (i.e. to `Classes/FluidDataSet`), PD and CLI, but I don't think there's a way in Max to link to another object and have alternate link text: what we need to have rendered is `<o>fluid.dataset~</o>`. 
+
+### Logging 
+
+Ah, I've already gone on about this a bit. Some effort was made to try and improve the quality of information given back. Notably: 
+1. running make ref will produce a `flucoma-makeref.log` file, with any warnings. These will still be written to `stderr` as well, unless the script is run with `--quiet`
+2. A problem before was that the docutils warnings weren't very helpful (partly because we process in fragments, so the line numbers were meaningless). I now try and record both the broad context, and retrieve the actual troublesome markup. 
+3. It's all a bit Heath-Robinson, in that I've had to wrap the logging framework in and around all these other packages. But, for all that it's not so obvious how it all works, it's not very intrusive: it can be altered, or turned off entirely very easily. 
+
+### Drivers 
+
+Contain all that is needed to account for the differences between our different rendering targets. It's all still a bit heterogenous and scrappy. Perhaps making the data structures more categorically clear with some hierarchy, and being more Pythonic about using `kwargs` would make it all seem less ad hoc.  
+
+## TODOs 
+
+* Test coverage is pretty woeful, partly because I'm getting near to the stuff that's hard to test, like rendering. 
+* There should be some authoring docs that detail the few bits of reStructuredText magic we use (and give a more comprehensive cookbook once we've transitioned to reST properly)
+* These code docs could obviously be much more extensive. One possibility would be to use Sphinx to manage these and host on the github pages for this repo (which might be an interesting test case ahead of documenting core)
+* The C++ remains mysterious to all but I 
+* Rendering control constraints is still a horror show, but less than it was 
+* All the code could be made more readable still 

doc/AmpGate.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest: Amplitude-based Gating Slicer
+species: slicer
 sc-categories: Libraries>FluidDecomposition
 sc-related: Guides/FluCoMa, Guides/FluidDecomposition
 see-also: BufAmpGate, AmpSlice, OnsetSlice, NoveltySlice, TransientSlice

doc/AmpSlice.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest: Amplitude-based Detrending Slicer
+species: slicer
 sc-categories: Libraries>FluidDecomposition
 sc-related: Guides/FluCoMa, Guides/FluidDecomposition
 see-also: BufAmpSlice, AmpGate, OnsetSlice, NoveltySlice, TransientSlice

doc/AudioTransport.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest: Interpolate between sounds
+species: transformer
 sc-categories: FluidManipulation
 sc-related: Classes/FluidBufAudioTransport
 see-also: 

doc/BufAmpGate.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest: Amplitude-based Gating Slicer for Buffers
+species: buffer-proc
 sc-categories: Libraries>FluidDecomposition
 sc-related: Guides/FluCoMa, Guides/FluidDecomposition
 see-also: AmpGate, BufAmpSlice, BufOnsetSlice, BufNoveltySlice, BufTransientSlice

doc/BufAmpSlice.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest: Amplitude-based Detrending Slicer for Buffers
+species: buffer-proc
 sc-categories: Libraries>FluidDecomposition
 sc-related: Guides/FluCoMa, Guides/FluidDecomposition
 see-also: AmpSlice, BufAmpGate, BufOnsetSlice, BufNoveltySlice, BufTransientSlice

doc/BufAudioTransport.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest: Interpolate between buffers
+species: buffer-proc
 sc-categories: FluidManipulation
 sc-related: Classes/FluidAudioTransport
 see-also: 

doc/BufChroma.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest: An histogram of pitch classes on a Buffer
+species: buffer-proc
 sc-categories: Libraries>FluidDecomposition
 sc-related: Guides/FluCoMa, Guides/FluidDecomposition, Classes/FluidBufMFCC
 see-also: Chroma, BufPitch, BufLoudness, BufMFCC, BufSpectralShape, BufStats

doc/BufCompose.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest: Buffer Compositing Utility
+species: buffer-proc
 sc-categories: Libraries>FluidDecomposition, UGens>Buffer
 sc-related: Guides/FluCoMa, Guides/FluidDecomposition, Classes/Buffer
 see-also:

doc/BufFlatten.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest: Flatten a multichannel buffer
+species: buffer-proc
 sc-categories: FluidCorpusManipulation
 sc-related: Classes/Buffer
 see-also:

doc/BufHPSS.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest : Buffer-Based Harmonic-Percussive Source Separation Using Median Filtering
+species: buffer-proc
 sc-categories : Libraries>FluidDecomposition, UGens>Buffer
 sc-related : Guides/FluCoMa, Guides/FluidDecomposition
 see-also: HPSS, BufSines, BufTransients

doc/BufLoudness.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest: A Loudness and True-Peak Descriptor on a Buffer
+species: buffer-proc
 sc-categories: Libraries>FluidDecomposition
 sc-related: Guides/FluCoMa, Guides/FluidDecomposition
 see-also: Loudness, BufPitch, BufMelBands, BufMFCC, BufSpectralShape, BufStats

doc/BufMFCC.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest: Mel-Frequency Cepstral Coefficients as Spectral Descriptors on a Buffer
+species: buffer-proc
 sc-categories: Libraries>FluidDecomposition
 sc-related: Guides/FluCoMa, Guides/FluidDecomposition, Guides/FluidBufMultiThreading, Classes/FluidBufMelBands
 see-also:

doc/BufMelBands.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest: A Perceptually Spread Spectral Contour Descriptor on a Buffer
+species: buffer-proc
 sc-categories: Libraries>FluidDecomposition
 sc-related: Guides/FluCoMa, Guides/FluidDecomposition, Classes/FluidBufMFCC
 see-also: MelBands, BufPitch, BufLoudness, BufMFCC, BufSpectralShape, BufStats

doc/BufNMF.yaml

@@ -7,6 +7,7 @@
 # (grant agreement No 725899).
 ---
 digest: Buffer-Based Non-Negative Matrix Factorisation on Spectral Frames
+species: buffer-proc
 sc-categories: Libraries>FluidDecomposition, UGens>Buffer
 sc-related: Guides/FluCoMa, Guides/FluidDecomposition, Classes/FluidNMFMatch, Classes/FluidNMFFilter
 see-also: NMFMatch, NMFFilter