dvc with gdrive service account

[eangius]

We are loving the potential DVC brings, but it is not clear from the documentation, discussions & issues how to properly setup a gdrive remote to be accessed via a service-account for our CI machines to automatically pull artifacts without having to interactively authenticate through the browser link.

We created the GDRIVE_CREDENTIALS_DATA environment variable with the JSON content (as a single line) of our google service account:

{
  "type": "service_account",
  "project_id": "<my_project_name>",
  "private_key_id": "<my_key_id>",
  "private_key": "-----BEGIN PRIVATE KEY-----\n<my_private_key>\n-----END PRIVATE KEY-----\n",
  "client_email": "<[email protected]>",
  "client_id": "<my_client_id>",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "<some_cert_url>"
}

We then invoke these commands:

dvc remote modify <myremote> gdrive_use_service_account true;
dvc remote modify <myremote> gdrive_service_account_user_email <[email protected]>;
dvc remote modify <myremote> gdrive_service_account_json_file_path /tmp/to_be_created_credentials.json;  # not /dev/null as this is to store temp session credentials
dvc pull --verbose;

NOTE: all instances of "<whatever>" above are placeholders for our real values.

And always get the error:

ERROR: failed to pull data from the cloud - GDrive remote auth failed with credentials in 'GDRIVE_CREDENTIALS_DATA'.
Backup first, remove or fix them, and run DVC again.
It should do auth again and refresh the credentials.

Details:: '_module'

When we instrument the gdrive.py#L203 source code, we notice the default client-id/secret are being used instead. Is this a bug or a mis configuration on our end? 🙏🏼

[eangius]

Seems like the GDriveFileSystem class gets instantiated with an incomplete config missing:

• gdrive_use_service_account
• gdrive_service_account_user_email
• gdrive_service_account_json_file_path

[eangius]

Any updates on this? Seems like a bug.

[shcheklein]

Hi @eangius !

Sorry for the delay. I was not able to reproduce this so far.

Seems like the GDriveFileSystem class gets instantiated with an incomplete config missing

The value comes from the DVC config to the remote class here

And it overrides default global PyDrive2 auth settings (here)[https://github.com/iterative/dvc/blob/1507a6ed99cb943d368beb39cd58a80fa750d85c/dvc/fs/gdrive.py#L224]

Details:: '_module'

what version of DVC are you using? could you check the latest one?

does it work for you locally?

[eangius]

The gdrive_* configs used to NOT be read, so we patched back then our values in that area of the code. Latest dvc release seems to be working! Thanks @shcheklein.