pyAirbyte in air gapped environment

[ziggekatten]

Trying to use Airflow & pyAirbyte to extract data from an internal API to an postgresql cache using docker containers. On my local machine it works fine, I build my container, do an pip install and everything is nice and dandy. However, it seems that the first time the image is used, it fetches some things from the repo, and when deploying the image in our air gapped environment this does not work. How do I install neccessary things beforehand to be able to use my custom API-source yaml generated by Airbyte builder? The URl that times out is: https://connectors.airbyte.com/files/registries/v0/oss_registry.json

Or is it just built in to connect to the URI no matter what? Is it possible to disable? I thougt "install_if_missing=False" would fix this.

My source def in Airflow:

       # Define a Postgres Cache and pass the necessary configuration
        pg_cache = PostgresCache(
            host=dbparams['host'],
            port=dbparams['port'],
            username=dbparams['login'],
            password=dbparams['password'],
            database=dbparams['db']
            )

        with open(os.path.join(os.getcwd(), 'plugins/airbyte/sources/api.yaml'), 'r') as file:
            source_manifest_dict: dict = yaml.safe_load(file)
        source = ab.get_source(
            'My custom API',
            config = apicreds,
            source_manifest=source_manifest_dict,
            streams = '*',
            install_if_missing=False
        )

[ziggekatten]

Answer myself here. Digging through the python code, I found the logic that checks the registry. The code have an built in feature to look at an local file if an environment variable was set, so one can get around this. I pulled the URL to an json-file, and added the environment variable to my airflow stack:

AIRBYTE_LOCAL_REGISTRY: '/filepath/oss-registry.json'

And hey presto! It works!

[aaronsteers]

Nice, @ziggekatten!

Another user had a similar question here:

• Offline mode #428 (comment)

As discussed in that issue, we (Airbyte) would be happy to accept a PR if there's another solution that would satisfy this use case.