Add structured config compilation #8098

fantix · 2024-12-10T22:48:18Z

Structured config values are Python objects from environment variables or (future) TOML config files. The compilation re- assembles the objects into ConfigOp ASTs and uses the static evaluation mechanism to generate verified config values.

This supersedes #8059 as a less-hacky way to validate config values.

As a side-effect, such config sources now support specifying EdgeQL values directly in a double-braces, for example:

env GEL_SERVER_CONFIG_cfg::session_idle_timeout="{{<duration>'8 minutes 32 seconds'}}"

Or even simpler:

env GEL_SERVER_CONFIG_cfg::session_idle_timeout='8 minutes 32 seconds'

(because the typecast is included by default)

This allows setting nested config objects with INSERT statements too:

env GEL_SERVER_CONFIG_cfg::email_providers="{{ { (insert cfg::SMTPProviderConfig { name := 'gmail'}) } }}"

Sample:

ops = compile_structured_config(
    {
        "cfg::Config": {
            "singleprop": "value",
            "memprop": 512,
            "durprop": "16s",
            "enumprop": "One",
            "multiprop": ["v1", "v2", "v3"],
            "listen_port": 5,
            "email_providers": [
                {"_tname": "cfg::SMTPProviderConfig", "name": "gmail"},
                {"_tname": "cfg::SMTPProviderConfig", "name": "outlook"},
            ],
            "sysobj": [
                {
                    "name": "1",
                    "obj": {
                        "_tname": "cfg::Subclass1",
                        "name": "aa",
                        "sub1": "aa",
                    },
                },
                {
                    "name": "2",
                    "_tname": "cfg::TestInstanceConfigStatTypes",
                    "memprop": 128,
                },
            ],
        },
    },
    spec=spec,
    schema=stdlib.stdschema,
    source="config file",
)
print(ops)

{
 'cfg::Config': immutables.Map({
  'durprop': SettingValue(name='durprop', value=<statypes.Duration 'PT16S'>, source='config file', scope=<ConfigScope.INSTANCE: 'INSTANCE'>, secret=False),
  'email_providers': SettingValue(name='email_providers', value=frozenset({cfg::SMTPProviderConfig(timeout_per_email=<statypes.Duration 'PT1M'>, security='STARTTLSOrPlainText', sender=None, name='outlook', port=None, timeout_per_attempt=<statypes.Duration 'PT15S'>, validate_certs=True, password=None, host=None, username=None), cfg::SMTPProviderConfig(timeout_per_email=<statypes.Duration 'PT1M'>, security='STARTTLSOrPlainText', sender=None, name='gmail', port=None, timeout_per_attempt=<statypes.Duration 'PT15S'>, validate_certs=True, password=None, host=None, username=None)}), source='config file', scope=<ConfigScope.INSTANCE: 'INSTANCE'>, secret=False),
  'enumprop': SettingValue(name='enumprop', value='One', source='config file', scope=<ConfigScope.INSTANCE: 'INSTANCE'>, secret=False),
  'listen_port': SettingValue(name='listen_port', value=5, source='config file', scope=<ConfigScope.INSTANCE: 'INSTANCE'>, secret=False),
  'memprop': SettingValue(name='memprop', value=<statypes.ConfigMemory '512B'>, source='config file', scope=<ConfigScope.INSTANCE: 'INSTANCE'>, secret=False),
  'multiprop': SettingValue(name='multiprop', value=frozenset({'v3'}), source='config file', scope=<ConfigScope.INSTANCE: 'INSTANCE'>, secret=False),
  'singleprop': SettingValue(name='singleprop', value='value', source='config file', scope=<ConfigScope.INSTANCE: 'INSTANCE'>, secret=False),
  'sysobj': SettingValue(name='sysobj', value=frozenset({cfg::TestInstanceConfigStatTypes(obj=None, name='2', durprop=None, memprop=<statypes.ConfigMemory '128B'>), cfg::TestInstanceConfig(obj=cfg::Subclass1(sub1='aa', name='aa'), name='1')}), source='config file', scope=<ConfigScope.INSTANCE: 'INSTANCE'>, secret=False)
 })
}

Structured config values are Python objects from environment variables or (future) TOML config files. The compilation re- assembles the objects into ConfigOp ASTs and use the static evaluation mechanism to generate verified config values. As a side-effect, such config sources now support specifying EdgeQL values directly in a double-braces, for example: env GEL_SERVER_CONFIG_cfg::session_idle_timeout \ = "{{<duration>'5s' * 2}}" Technically, this allows setting nested config objects with INSERT statements too.

elprans · 2024-12-12T22:10:22Z

Technically, this allows setting nested config objects with INSERT statements too.

Can you give an example of how that might look?

fantix · 2024-12-12T22:30:19Z

This works:

env GEL_SERVER_CONFIG_cfg::email_providers="{{ { (insert cfg::SMTPProviderConfig { name := 'gmail'}) } }}" edb server

It's compiled into such AST:

configure INSTANCE insert cfg::SMTPProviderConfig{ name := 'gmail' };

Then it's statically evaluated into:

immutables.Map(
  {'email_providers': SettingValue(
    name='email_providers',
    value=frozenset(
      {cfg::SMTPProviderConfig(
        host=None,
        sender=None,
        port=None,
        timeout_per_attempt=<statypes.Duration 'PT15S'>,
        password=None,
        name='gmail',
        validate_certs=True,
        username=None,
        timeout_per_email=<statypes.Duration 'PT1M'>,
        security='STARTTLSOrPlainText'
      )}
    ),
    source='environment variable', scope=<ConfigScope.INSTANCE: 'INSTANCE'>,
    secret=False)
  }
)

msullivan

Looks good broadly, but I have some concerns we need to figure out about the config object situtation

edb/ir/staeval.py

msullivan · 2024-12-12T23:37:30Z

edb/server/compiler/compiler.py

+    ) -> dict[str, immutables.Map[str, config.SettingValue]]:
+        # XXX: only config in the stdlib is supported currently, so the only
+        # key allowed in objects is "cfg::Config". API for future compatibility
+        if list(objects) != ["cfg::Config"]:


Should it arguably be cfg::AbstractConfig?

Yes? Though, it's a bit weird for the user to write [cfg::AbstractConfig] in the TOML config file.

Fwiw using cfg::Config here is following this line which was added before we even had cfg::AbstractConfig and never got updated.

msullivan · 2024-12-13T00:11:14Z

tests/test_server_compiler.py

+    def _test_compile_structured_config(
+        self,
+        values: dict[str, Any],
+        *,
+        source: str = "config file",
+        **expected: Any,
+    ) -> dict[str, config.SettingValue]:
+        result = self.compiler.compile_structured_config(
+            {"cfg::Config": values}, source=source
+        )
+        rv = dict(result["cfg::Config"])
+        for name, setting in rv.items():
+            self.assertEqual(setting.name, name)
+            self.assertEqual(setting.scope, config.ConfigScope.INSTANCE)
+            self.assertEqual(setting.source, source)
+        self.assertDictEqual({k: v.value for k, v in rv.items()}, expected)
+        return rv


Could we get an end-to-end test, also?

One thing I am concerned about is how INSERTed objects specified via structured config will interact with objects inserted manually via CONFIGURE INSTANCE INSERT ....

The best option is for the sets of objects to get merged. I spent about a day trying to make that work, and came to the conclusion that it would probably take me closer to a week, and would be pretty hairy. This is why I introduced EDGEDB_MAGIC_SMTP_CONFIG, so we wouldn't need to handle it.

If manually inserted objects just fully override, that is probably acceptable behavior (though it wouldn't work for our SMTP config case, so we would still need EDGEDB_MAGIC_SMTP_CONFIG), but I think it will take a little care to make that work properly also. (If you INSERT an object and then RESET it away, probably the structured config one should come back? But I think as implemented it won't quite work.)

Anyway, I don't think this really blocks us, but I think we might need to disable object config for now, and stick with EDGEDB_MAGIC_SMTP_CONFIG for the case we need. (We should probably use the structured parser for EDGEDB_MAGIC_SMTP_CONFIG though!)
I think getting objects to work well enough to cover that case for us will be a decent amount of extra work, and not a priority yet.

Right, I also left 2 similar questions in the previous PR

Added a switch to turn configure ... insert off for now. End-to-end tests for non-insert configs are covered by the env var static config tests now; the upcoming TOML config file PR will include more e2e tests.

We should probably use the structured parser for EDGEDB_MAGIC_SMTP_CONFIG

Fixed!

edb/server/main.py

msullivan · 2024-12-13T19:07:48Z

edb/server/main.py

            tenant=tenant,
            use_monitor_fs=args.reload_config_files in [
                srvargs.ReloadTrigger.Default,
                srvargs.ReloadTrigger.FileSystemEvent,
            ],
            net_worker_mode=args.net_worker_mode,
        )
+        magic_smtp = os.getenv('EDGEDB_MAGIC_SMTP_CONFIG')


Did you do a manual test of this, since apparently I half-assed it and didn't write one

Yes, manually tested ✅

fantix force-pushed the compile-structured-config branch 2 times, most recently from d8b875e to 3eff4c4 Compare December 12, 2024 16:53

fantix changed the base branch from master to bool-staeval December 12, 2024 16:53

Base automatically changed from bool-staeval to master December 12, 2024 17:25

fantix added 2 commits December 12, 2024 13:24

Use new API to validate env var config

621be4f

fantix force-pushed the compile-structured-config branch from 3eff4c4 to 12d3a25 Compare December 12, 2024 18:24

fantix marked this pull request as ready for review December 12, 2024 19:01

fantix changed the title ~~[WIP] Add structured config compilation~~ Add structured config compilation Dec 12, 2024

fantix requested review from msullivan and elprans December 12, 2024 19:01

Drop now-unused export

89b2809

msullivan reviewed Dec 13, 2024

View reviewed changes

fantix added 2 commits December 13, 2024 10:57

CRF: turn off inserting nested ConfigObject and add type check

9de684a

Use the new API for EDGEDB_MAGIC_SMTP_CONFIG

04af3be

fantix requested a review from msullivan December 13, 2024 17:38

msullivan approved these changes Dec 13, 2024

View reviewed changes

crf: drop compat patch

a2dc673

fantix merged commit e80225e into master Dec 13, 2024
23 checks passed

fantix deleted the compile-structured-config branch December 13, 2024 20:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add structured config compilation #8098

Add structured config compilation #8098

fantix commented Dec 10, 2024 •

edited

Loading

elprans commented Dec 12, 2024

fantix commented Dec 12, 2024 •

edited

Loading

msullivan left a comment

msullivan Dec 12, 2024

fantix Dec 13, 2024

msullivan Dec 13, 2024

msullivan Dec 13, 2024

fantix Dec 13, 2024

fantix Dec 13, 2024

fantix Dec 13, 2024

msullivan Dec 13, 2024

fantix Dec 13, 2024

Add structured config compilation #8098

Add structured config compilation #8098

Conversation

fantix commented Dec 10, 2024 • edited Loading

elprans commented Dec 12, 2024

fantix commented Dec 12, 2024 • edited Loading

msullivan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fantix commented Dec 10, 2024 •

edited

Loading

fantix commented Dec 12, 2024 •

edited

Loading