Configuration schema registry

[jacobcook1995]

So, I am in the process of writing a system to read in and validate config files (see issue #63). I am currently wondering how best to set up modular loading of config files. This is needed so that we can still validate config files in cases where we wanted to run a limited simulation with only particular modules. It is also potentially important if we developed alternative module implementations which would likely require different configuration parameters and data.

I've discussed this already with @davidorme, and we came up with the following plan to address it. Each module would have a separate json schema defined in their respective __init__.py. These would then all be added to a registry and decorated, so that the specific module schema could be accessed as e.g. SCHEMA_REGISTRY['plant'].

The core module always has to be configured, and so has to contain a list of the other modules to be configured. We use this list to find the other schema to validate configuration input against.

We were basically wondering @alexdewar @dalonsoa whether this seems like a sensible approach to you?

[davidorme]

This follows on from the suggestion for the grid (#28) . Basically, a system we will use that also permits users to add a new module with configuration by including in __init__.py:

@register_schema('new_module_name')
def module_schema():
    json_schema = """JSON SCHEMA HERE"""
    return json_schema

And then core/config.py would establish a registry so that config parsing can then validate TOML files containining:

[new_module_name.vital_config]
critical_value = 1

That might also make it easy (I think) to maintain and test parallel module implementations - you could have configs using plants and plants_modified within the same codebase. Duplicate the plants module, tweak the config name (or it could be read automatically from the module path?) and then have two parallel implementations which you can switch between in config rather than having to maintain different package builds where plants works differently.

[davidorme]

Another thing this should definitely do is be able to take a set of modules and write out a default basic config file using the JSON schemas for those modules. So at the command line:

vr --build_default_config plants,animals,soil,abiotic > default_config.toml

[alexdewar]

I've put some thoughts about this below, but the tl;dr of it is that I think you shouldn't be working on this feature right now. In essence, you don't need config files right now, because your code doesn't do anything, so there's nothing to configure!

If I were you, I would work on getting something working on the simulation front first and foremost so that you have some idea of what the Python API will look like. Once you've done that, you'll know what your data structures look like from Python's perspective and that will inform how you want to represent them in some other format (json, toml etc.) -- that is, if you really need to do that at all, which I'm not convinced about.

Although I'm obviously not as well versed in the project's goal as you guys, my guess is that the most sensible way to run your simulations, at least for the first while, will be programmatically from Python itself, e.g. you'll just write scripts like so:

import vr

PARAMETER = 10
SOME_OTHER_PARAMETER = 5

vr.run_simulation(parameter=PARAMETER, some_other_parameter=SOME_OTHER_PARAMETER)

You need to be able to configure and run your simulations from Python (even if this functionality is only an internal API in the end); you could run simulations from a command-line program which parses config files. Given that the second of these depends on the first, my advice would be to do the first one first and then consider doing the second.

Expressing your parameters from within Python itself is much simpler and cleaner. Another advantage is that your linter/CI system can check for e.g. type errors for you; no schemas needed.

If you desperately want to load parameters from an external config file, that's dead easy with vanilla Python:

import json, vr

with open('config.json', 'r') as file:
    config = json.load(file)

vr.run_simulation(**config)

From the Zen of Python "Simple is better than complex.": https://peps.python.org/pep-0020/

Am I missing something here?

[davidorme]

Possibly only missing my predisposition for premature optimisation 😄

However, there are some immediate barriers to progress that this addresses as well as reflecting longer term thoughts about the user experience. None of those are set down anywhere, so trying to unpack them:

• Although it will obvious be run programatically, I can see a lot of users preferring to run as a command line program. The UX is then: install python and virtual-rainforest and then you can just get up and running use example config files and run vr --config /path/to/dir --output /path/to/outputs. Not, 'and then you need to learn to program in Python'. So having that in mind from the start seems valuable.

• I think this package is going to have an unusually complex configuration, so having a well structured config tree structure is vital. vr.run(configObject) is a much more likely API and - even without thinking about writing config files and validation - this discussion is about forcing us to adopt a clear configuration structure from the start.

• That config structure is actually a barrier to writing current scientific code. The initial obvious cases are data input and grid structure - those have to be defined somewhere and coordinated across modules. For example, if modules wants access to temperature data, where does it come from? It should only be one place that is accessed by all modules. See the parallel issue on data configuration and loading at Adding design notes on core data setup and configuration #65.

• Type errors are not the only thing we want to validate. Config options may have bounds etc and the JSON schema provides a well-defined way to document that. That isn't to say you couldn't do that within methods but methods only validate when they are used.

• And hence, another thing we're considering here is validating the config as much as possible before simulation. We'd been considering a two stage process that allows an initial validation and helpful guidance on config errors. That avoids at least some of the UX of running the program for 2 hours and then getting "Error: disturbance.logging.selectively_log: proportion must be in [0,1] not 96".

  So use would be something like

from virtual_rainforest.core import VirtualRainforest
# instantiate a vr object using the config 
# - a single file, a list of files, a directory of files or a dictionary
# - fails if the config is incorrect
vr = VirtualRainforest(config: Union[str, list[str], dict]]) 

# run it
vr.run()

• That isn't to say validation couldn't be done within functions, methods and classes and there is a real issue here about what gets validated where. We could easily end up violating DRY by having to update lists of acceptable options in JSON schema and within objects.
• We could potentially use these schema to power a standard validate function within callables, moving all validation to be defined by a single source and centralising a lot of boiler plate validation within functions. That sounds a bit premature though!

[davidorme]

I guess this is classic 🐔 and 🥚. Our code can't do anything without being configured.

[alexdewar]

Possibly only missing my predisposition for premature optimisation smile

However, there are some immediate barriers to progress that this addresses as well as reflecting longer term thoughts about the user experience. None of those are set down anywhere, so trying to unpack them:

• Although it will obvious be run programatically, I can see a lot of users preferring to run as a command line program. The UX is then: install python and virtual-rainforest and then you can just get up and running use example config files and run vr --config /path/to/dir --output /path/to/outputs. Not, 'and then you need to learn to program in Python'. So having that in mind from the start seems valuable.

If you want users to be able to run simulations from the command-line with arguments or a config file, that's all well and good, but what I'm saying is that those are effectively the outermost parts of your program/library, and it would be better to start from the inside and work out. Once you have started writing simulation code, you'll a) have a better idea of what aspects of this simulation you want to be user-controllable and b) you'll know roughly what kind of interface you want to be able to expose to the user (e.g. these parameters should be customisable). This will need to be accessible from within Python: you'll need classes, functions etc. Then when you know what this broad structure looks like you can think about how to represent that as e.g. a config file. When you do need to do this, you can do it in just a few lines of code, as I showed above, so it's really early to be thinking about this problem!

This isn't a chicken/egg problem AFAICS. This is a "tiling your roof before the house is built problem". The danger in doing it this way round is that you might well find that the roof you've spent ages tiling doesn't nicely fit onto the rest of the structure.

• I think this package is going to have an unusually complex configuration, so having a well structured config tree structure is vital. vr.run(configObject) is a much more likely API and - even without thinking about writing config files and validation - this discussion is about forcing us to adopt a clear configuration structure from the start.
• That config structure is actually a barrier to writing current scientific code. The initial obvious cases are data input and grid structure - those have to be defined somewhere and coordinated across modules. For example, if modules wants access to temperature data, where does it come from? It should only be one place that is accessed by all modules. See the parallel issue on data configuration and loading at Adding design notes on core data setup and configuration #65.

There's nothing wrong with having a single place with your configuration (though I dislike the idea of reading and parsing a config file over and over again for every module), but why can't you just pass around a dict containing all the parameters you're using for now? Yes you need to figure out how to represent your data input and yes you need to figure out how to represent temperatures, but you need to know how these structures are going to be represented as objects/lists etc. within Python. There's really no getting around that. Trying to figure out how you'll represent them in e.g. JSON first just complicates matters. Once you know what these structures are, you can serialise/deserialise them with dozens of well-tested standard Python libraries. I'd strongly advise you not to go down the route of writing your own library for reading/writing arbitrary Python objects when there are so many alternatives out there!

• Type errors are not the only thing we want to validate. Config options may have bounds etc and the JSON schema provides a well-defined way to document that. That isn't to say you couldn't do that within methods but methods only validate when they are used.
• And hence, another thing we're considering here is validating the config as much as possible before simulation. We'd been considering a two stage process that allows an initial validation and helpful guidance on config errors. That avoids at least some of the UX of running the program for 2 hours and then getting "Error: disturbance.logging.selectively_log: proportion must be in [0,1] not 96".

Aha, I think I finally know where you're coming from 😉! Yes, this is a real problem; thoughts below.

So use would be something like

from virtual_rainforest.core import VirtualRainforest
# instantiate a vr object using the config 
# - a single file, a list of files, a directory of files or a dictionary
# - fails if the config is incorrect
vr = VirtualRainforest(config: Union[str, list[str], dict]]) 

# run it
vr.run()

• That isn't to say validation couldn't be done within functions, methods and classes and there is a real issue here about what gets validated where. We could easily end up violating DRY by having to update lists of acceptable options in JSON schema and within objects.
• We could potentially use these schema to power a standard validate function within callables, moving all validation to be defined by a single source and centralising a lot of boiler plate validation within functions. That sounds a bit premature though!

I think the issue I've had here in understanding what the motivation for all this was that you proposed a solution (custom config-file handling code) rather than stating the problem it's trying to solve (validation that parameters are all acceptable before the simulation starts). It could also be that I didn't read what you initially wrote closely enough -- sorry for confusion.

I would say that the problem with using JSON schemas is precisely that it will probably involve violating DRY, as I suspect you'll end up expressing constraints for parameters both in Python and within the schemas and these will need to be kept in sync.

Here's an alternative proposal. When each of your modules is loaded, have your core code invoke a method in their respective __init__.py files called load_module(), that takes your config object/dict and validates it, then does any other initial steps needed to initialise the module.

A silly trivial example could look like this:

def validate_temperature(temp):
    if temp < 0.0:
        raise ValueError("Too cold")
    elif temp > 100.0:
        raise ValueError("Too hot")

def load_module(config):
    validate_temperature(config["temperature"])

    # ...(do other setup)

def run_module(temp):
    # We do this as a sanity check
    validate_temperature(temp)

   # ...(do something else)

Yes, it's silly, but you get the general idea: abstract constraint-checking away into separate functions. If you want more complex error handling than is currently there (e.g. checking that the "temperature element actually exists) you can just add another check, throwing a ValueError if it fails. The idea would be that the calling function can do something sensible with errors (e.g. saying which parameter and module are the problem).

[davidorme]

Points taken! Just to clarify a few things:

That config structure is actually a barrier to writing current scientific code. The initial obvious cases are data input and grid structure - those have to be defined somewhere and coordinated across modules. For example, if modules wants access to temperature data, where does it come from? It should only be one place that is accessed by all modules. See the parallel issue on data configuration and loading at #65.

There's nothing wrong with having a single place with your configuration (though I dislike the idea of reading and parsing a config file over and over again for every module),

Yes - this was never the intention. Configuration validation happens once and then that configuration object is passed around as a single consistent dictionary of parameterisation.

but why can't you just pass around a dict containing all the parameters you're using for now? Yes you need to figure out how to represent your data input and yes you need to figure out how to represent temperatures, but you need to know how these structures are going to be represented as objects/lists etc. within Python.

Yes - and for some things we know this already and need a consistent way to load and pass them. There will be np.array objects for a wide range of core forcing variables associated with the spatial grid of the simulation. We need a common central structure for all of that - and figuring out to load them from file, sanity check and provide a consistent reference framework for those shared core inputs is really where that discussion started. That's what I meant about chicken and egg - we need some of this now.

There's really no getting around that. Trying to figure out how you'll represent them in e.g. JSON first just complicates matters. Once you know what these structures are, you can serialise/deserialise them with dozens of well-tested standard Python libraries. I'd strongly advise you not to go down the route of writing your own library for reading/writing arbitrary Python objects when there are so many alternatives out there!

We wouldn't disagree, but the design notes explain why we're looking at these options: we need a human-readable, well structured format that users can edit to change the configuration of the model. This isn't just serialise/deserialise - users aren't going to edit JSON or pickle 😄

Here's an alternative proposal. When each of your modules is loaded, have your core code invoke a method in their respective init.py files called load_module(), that takes your config object/dict and validates it, then does any other initial steps needed to initialise the module.

Actually, that's sort of what we have in mind, but generalising it beyond variable specific functions.

def validate(config):

    # clearly missing some hasattr() and getattr()
    for paths in config:
         if config[path]  < schema[path]["minimum"]:
            raise ValueError(f"Value {path} below minimum")
   
def load_module(config):
    validate(config)

    # ...(do other setup)

def run_module(temp):
    # We do this as a sanity check
    validate(config)

This doesn't actually violate DRY - we have a single source of validity requirements in schema and a single boiler plate validator. The reason we were looking at JSON schema is because the jsonschema package wraps a lot of those validation options and might make the validate function into a one liner . It doesn't really matter whether the requirements are in a JSON schema file or a dictionary.

[dalonsoa]

I think you have covered pretty much every point on both sides of the argument, so I don't think I can add much. Just in case, here you have my opinion:

• It is important to spend as much time as needed thinking on you data model and how input/output would look like. This drives the correct choice of tools for the job. Using JSON and JSON Schemas looks sensible, although for a bit more human readable approach I would use YAML for the input files, which is also compatible with JSON Schemas (and let you add explanatory comments).
• No matter how much you think on this, you will need to revisit your initial decisions once and again and be ready to change them until the software is somewhat mature. Iteration is the basis of Agile development, so forget about making decisions now that won't require changing in the future.
• As I'm a big fan of registries, as you might have noticed, I would decorate all validator functions, add them to a registry and then, when launching the simulation, run all the validators in the registry on the config. This enables easily expanding the validators and does not make any assumption about where to put them. Not totally sure how this integrates with JSON Schemas because I've never actually used them, but seems like the most flexible approach.

[davidorme]

Thanks - that's a really useful overview.

• We're planning to use TOML for the actual config files and just using the JSONSchema specification system internally to set up the config validation scheme.
• We have no illusions that these structures won't change 😄
• Essentially JSONSchema allows us to create a document (really just a dictionary) that allows us to define the config structure and then add a range of constraints on values at the tips. It then has a single function that compares a loaded config dictionary (from TOML, YAML, JSON - don't think it matters) against the schema structure and constraints.

[alexdewar]

@davidorme Sorry about slow reply! Yes I think @dalonsoa has summarised it well.

If we go back to your snippet though:

def validate(config):

    # clearly missing some hasattr() and getattr()
    for paths in config:
         if config[path]  < schema[path]["minimum"]:
            raise ValueError(f"Value {path} below minimum")

My main concern is what would happen if you decided you wanted to add e.g. a maximum to your parameter as well. Presumably you'd have to update both the python code and the JSON schema, else you'd have a bug? Other than that, I'd personally be reluctant to introduce what is essentially a separate language (i.e. JSON schemas) just for checking parameters when you can express everything in python anyways (and in fact you can express plenty of things you couldn't with a JSON schema....).

Just a thought though!

[davidorme]

In this specific example, JSONSchema describes a really broad set of possible constraints, and the JSONSchema package already implements a validate(config, schema) command that implements checking all those constraints with nice error messages. Essentially a Schema is just a python dictionary, and the implementation in python just provides a concise way to document those constraints. So maximum is covered 😄

But... more broadly. There are absolutely going to be configuration settings that will be hard / awkward to express using JSONSchema (depends on, etc) and which will need to be checked at run time. I think this is primarily about a first sanity check.