git-sync on airflow does not support SSH authentication

[raam86]

The current object only supports the username + password authentication method:

airflow-operator/pkg/apis/airflow/v1alpha1/utils.go

Line 307 in 1fbc02d

if sp.DAGs.Git.CredSecretRef != nil {

in order to fully support git-sync we need to 1. set GIT_SYNC_SSH to true and map a mounted volume that points to the ConfigMap: https://github.com/kubernetes/git-sync/blob/9ceb61f7947fbe463b1cc6e9ae5d719f5d8eebd2/docs/ssh.md#step-3-configure-git-sync-container

[barney-s]

Would you have time to take this up ? @raam86
I can help you with the setup and tests

[raam86]

After some more research I opted for GCS sidecar container, I think it's the best of all worlds. Basically lifted your setup into the helm/charts/incubator

[barney-s]

Thanks. What did you mean by lifted you setup into helm/charts ?

If you are using both i would love to hear you feedback on helm/charts vs the operator.

[raam86]

I initially used the operator and got quite excited but was stuck when trying to customize it. I specifically wanted to have the gcs sync as a sidecar instead of an init container and it was easier to do using the helm chart. Since I realized I am going to do work anyway I wanted it to be less google specific so I opted for the helm chart. the helm chart also has more users, more support and is more standard

[barney-s]

Ah i see. GCS sync is a side-car. The default value of .spec.dags.gcs.once is false.
Which helm chart are you using ?

[raam86]

The one that just became stable. You are definitely right about the defaults. I think that what actually happened was that I tried using git sync realized I can’t opted for the helm chart and eventually opted for gcs. Sorry about the confusion. But the underlying reason for continuing using the helm chart instead of the operator is that the helm chart uses standard and available kubernetes concepts while the operator is minting bew ones I don’t have lots of incentive to learn

[dmateusp]

hi there, since this is the first Google result when looking for SSH auth for git-sync on Airflow just wanted to let you know I've been working on adding this to the Airflow project:

https://issues.apache.org/jira/browse/AIRFLOW-3918
apache/airflow#4777