Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3Metadata should lazily load schemas #86

Open
adrdc opened this issue Nov 10, 2021 · 0 comments
Open

S3Metadata should lazily load schemas #86

adrdc opened this issue Nov 10, 2021 · 0 comments
Assignees

Comments

@adrdc
Copy link
Collaborator

adrdc commented Nov 10, 2021

in ctor for S3Metadata object (ConnectorMetadata), it gets table config. that process involves looking for metadata search keys from OBS/ECS, requesting data from schema registry, loading schemas from disk, etc.

however this is only needed on the coordinator. this info is not accessed on workers. so N workers will be pulling this same info down all to just be not used.

@Inject
public S3Metadata(S3ConnectorId connectorId,
                  Supplier<Map<SchemaTableName,S3Table>> s3TableDescriptionSupplier,
                  S3ConnectorConfig s3ConnectorConfig) {
	...
        this.tableDescriptions = s3TableDescriptionSupplier.get();
        ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants