[CT-458] Catalog queries should filter on specific relations in busy schemas

[jtcohen6]

Related:

• [CT-171] [CT-60] [Bug] It takes too long to generate docs (BigQuery) dbt-bigquery#115
• [CT-385] Limit catalog generation to specific relations dbt-spark#300

To enable this, _get_catalog_schemas would need to return a specific set of tables, rather than a set of database-schema combos (SchemaSearchMap):

dbt-core/core/dbt/adapters/base/impl.py

Lines 313 to 338 in 7f953a6

	def _get_catalog_schemas(self, manifest: Manifest) -> SchemaSearchMap:
	"""Get a mapping of each node's "information_schema" relations to a
	set of all schemas expected in that information_schema.
	There may be keys that are technically duplicates on the database side,
	for example all of '"foo", 'foo', '"FOO"' and 'FOO' could coexist as
	databases, and values could overlap as appropriate. All values are
	lowercase strings.
	"""
	info_schema_name_map = SchemaSearchMap()
	nodes: Iterator[CompileResultNode] = chain(
	[
	node
	for node in manifest.nodes.values()
	if (node.is_relational and not node.is_ephemeral_model)
	],
	manifest.sources.values(),
	)
	for node in nodes:
	relation = self.Relation.create_from(self.config, node)
	info_schema_name_map.add(relation)
	# result is a map whose keys are information_schema Relations without
	# identifiers that have appropriate database prefixes, and whose values
	# are sets of lowercase schema names that are valid members of those
	# databases
	return info_schema_name_map

Given that we'll need to interpolate table names into actual queries, maybe we could have a rule like: If a database-schema has >100 objects in it, just catalog the entire schema; any less, pass the specific object names as filters.

We see this issue crop up on several adapters, including most recently Redshift. Catalog generation is different from cache generation in two ways:

• The cache needs an exhaustive list of the objects in a schema in order to reliably say whether the object exists or not. The stakes are lower for cataloging.
• We catalog source tables, which commonly live in shared schemas, and in ways that dbt developers cannot easily control/change

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

[jeremyyeo]

Reopening on behalf of enterprise customer.

[graciegoheen]

Closing this out, since we completed the dbt-core implementation!

Adapter implementations are ongoing :)

[aerielsoriano]

Reopening on behalf of enterprise customer.