Description
When setting flattening_enabled to true then running meltano select tap-stackexchange --list --all an error message returns without a needed change expressed.
Example of the error:
Cannot list the selected attributes: Catalog discovery failed: command ['/meltano/.meltano/extractors/tap-stackexchange/venv/bin/tap-stackexchange', '--config', '/meltano/.meltano/run/tap-stackexchange/tap.07702730-c1d8-433b-ab2c-92b6271df905.config.json', '--discover'] returned 1
I found a bread crumb by going back and running meltano invoke tap-stackexchange --about, which returned this error:
Traceback (most recent call last):
File "/meltano/.meltano/extractors/tap-stackexchange/venv/bin/tap-stackexchange", line 8, in <module>
sys.exit(TapStackExchange.cli())
File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 467, in cli
tap = cls( # type: ignore # Ignore 'type not callable'
File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/singer_sdk/tap_base.py", line 87, in __init__
self.mapper = PluginMapper(
File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/singer_sdk/mapper.py", line 540, in __init__
self.flattening_options = get_flattening_options(plugin_config)
File "/meltano/.meltano/extractors/tap-stackexchange/venv/lib/python3.8/site-packages/singer_sdk/helpers/_flattening.py", line 34, in get_flattening_options
return FlatteningOptions(max_level=int(plugin_config["flattening_max_depth"]))
KeyError: 'flattening_max_depth'
I took a guess at setting flattening_max_depth config and it fixed the issue, but there is nothing that I can find stating flattening_max_depth is required when setting flattening_enabled.
A possible solution instead of updating the error handling would be to remove flattening_enabled altogether as a config and have it's state set by the flattening_max_depth. For instance, if flattening_max_depth is set then flattening is enabled to that depth, otherwise flattening is off.