REST endpoints send incomplete WWW-Authenticate header on 401

[yarikoptic]

ATM (same example as in #1824 ):

❯ wget -S 'https://api.dandiarchive.org/api/dandisets/000403/versions/draft/assets/0cb481e5-b979-4eed-8ea4-9cd0d78276a3/download/' 2>&1 | grep Auth
  Www-Authenticate: Bearer realm="api"

and for that (Bearer) scheme I see https://datatracker.ietf.org/doc/html/rfc6750#section-2.1 listing

Authorization: Bearer mF_9.B5f-4.1JqM

whenever we seems to be using token not Bearer:

❯ rm -f dataset_description.json; curl -J -O -L -X 'GET' --silent -H "Authorization: token $DANDI_API_KEY" 'https://api.dandiarchive.org/api/dandisets/000403/versions/draft/assets/0cb481e5-b979-4eed-8ea4-9cd0d78276a3/download/' ; ls -ld dataset_description.json
-rw------- 1 yoh yoh 84 Jan 24 13:05 dataset_description.json

and if I specify as Bearer -- no good:

❯ rm -f dataset_description.json; curl -J -O -L -X 'GET'  -H "Authorization: Bearer $DANDI_API_KEY" 'https://api.dandiarchive.org/api/dandisets/000403/versions/draft/assets/0cb481e5-b979-4eed-8ea4-9cd0d78276a3/download/' ; ls -ld dataset_description.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: Remote filename has no length
  0    58    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (23) Failure writing output to destination
ls: cannot access 'dataset_description.json': No such file or directory

In DataLad we are using Bearer keyword there and hence our implementation for this authorization scheme (https://github.com/datalad/datalad/blob/HEAD/datalad/downloaders/http.py#L401) is not working. It works if I change in our code "Bearer" to "token" but it seems to be invalid and I think the quickest fix would be for dandi-archive to add support for having "Bearer" specified in addition to "token" and slowly (after we have our dandi-cli etc) deprecated "token".

[waxlamp]

If you look at the DRF documentation for authentication you'll see that this middleware is what implements the header scheme with the word Token rather than Bearer.

Since we're relying on a well-established third party library for this, I don't think this can be considered a bug, especially not within our codebase.

To more closely address your problem: can you simply create a mechanism to authenticate to DANDI using the Token scheme? (This would be a "problem" for any API implemented with DRF, so I would be surprised if DataLad cannot solve this problem on its end.)

[yarikoptic]

My summary of further discussion on slack: agreement that there is a bug somewhere that we are responding with

Www-Authenticate: Bearer realm="api"

while not following OAuth 2.0 for bearer token authentication. DRF and page linked says it should be the

Www-Authenticate: Token

in such a case. Is it the location where we specify the authenticator? https://github.com/dandi/dandi-archive/blob/HEAD/dandiapi/settings.py#L56

[yarikoptic]

FWIW, if I am correct about that location about possible auth schemes, then may be to support also the Authorization: Bearer KEY we just need to add such a simple custom subclass into possible authentication schemes (from https://github.com/encode/django-rest-framework//commit/ffdac0d93619b7ec6039b94ce0e563f0330faeb1#diff-fe702eb4ce0f41059045e995535201a805bb92a7fbb162d4036087626c0a6c85R38):

class CustomKeywordTokenAuthentication(TokenAuthentication):
    keyword = 'Bearer'

PS edits

• I still have no clue what part of the django sandwich here responds with WWW-Authenticate: Bearer, heh
• github seems to allow both "bearer" and "token": https://docs.github.com/en/rest/authentication/authenticating-to-the-rest-api?apiVersion=2022-11-28

[yarikoptic]

ok, I have a recipe on how to proceed:

• use datalad from https://github.com/yarikoptic/datalad/tree/enh-token until Add the "http_token" authentication mechanism which provides 'Authentication: Token {TOKEN}' header datalad/datalad#7551 is reviewed/merged/released
• for every private/embargoed dandiset provide dandi provider config and initialize datalad special remote

this script does it and shows all needed steps

#!/bin/bash

cd "$(mktemp -d ${TMPDIR:-/tmp}/dl-XXXXXXX)"
set -eux

git init
git annex init

# generate config to ship along

mkdir -p .datalad/providers

cat >| .datalad/providers/dandi.cfg << EOF
[provider:dandi]
url_re = https?://api\.dandiarchive\.org/api/.*
authentication_type = http_token
credential = dandi

[credential:dandi]
type = token
EOF

git add .datalad/providers/dandi.cfg; git commit -m 'Added dandi provider config' 

git annex initremote datalad type=external externaltype=datalad encryption=none autoenable=true

git annex addurl  --file dataset_description.json https://api.dandiarchive.org/api/dandisets/000403/versions/draft/assets/0cb481e5-b979-4eed-8ea4-9cd0d78276a3/download/

cat dataset_description.json

git annex whereis

[waxlamp]

I have opened a PR to fix this in DRF.

However, I'm going to close this as unplanned for the following reasons:

• @yarikoptic has found a workaround
• this is a low-impact bug to fix (given that a workaround was found)
• it's not clear whether/when my PR will be accepted

@yarikoptic, let me know if there is more to discuss here.