Support discovery/indexing datasets from forgejo-aneksajo instances

[yarikoptic]

https://codeberg.org/matrss/forgejo-aneksajo

known instances:

• https://hub.datalad.org
• https://atris.fz-juelich.de
• https://hub.trr379.de/
• ...

at worst could be all repositories similar to GIN I guess.

Related (also to interegrate/use API, has pointers):

• forgejo support con/tinuous#194

[matrss]

Another instance: https://atris.fz-juelich.de/

Since Forgejo is a descendant of Gitea, which is a descendant of Gogs, I wouldn't be surprised if the existing code for GIN can be easily adapted to index forgejo-aneksajo instances as well.

[yarikoptic]

indeed. Likely could be started as what we have for GIN already -- getting all of them but may be adding some extra sensing/filtering later on to blacklist/not consider some.

[yarikoptic]

whenever GIN had really in mind sharing data, I think forgejo-aneksajo are growing into a more generic "alternatives to github". So if we are to implement their indexing, we might want to add some kind of "filtering" based on sensing on whether they have e.g. git-annex branch.

[jwodder]

@yarikoptic What instance(s) should be tracked?

[yarikoptic]

could start from the first but ideally all 3, since all "related" and should contain datalad or at least git-annex repos.

[jwodder]

@yarikoptic I can't find any documentation on forgejo-aneksajo's or hub.datalad.org's supported search query syntax. There also doesn't seem to be a way to register on hub.datalad.org so that I can get an API key.

[jwodder]

@yarikoptic How exactly do you want me to query forgejo-aneksajo instances for Datalad repositories? What data should I collect?

[yarikoptic]

• you have account now now on hub.datalad.org
• "how" is to be figured out. From above - might be as on GIN, no figuring out but going through the full listing.
  • a public instance of forgejo (without git-annex support) is http://codeberg.org/
  • high level api doc is at https://forgejo.org/docs/latest/user/api-usage/
  • https://hub.datalad.org/api/v1/repos/search/ seems to work as on GIN, might so might indeed just reuse the same interface (ATM doesn't even require authentication but I would not count on that -- @mih or I might lock it down if some bad actor starts to abuse API)

[jwodder]

@yarikoptic It looks like listing all the repos via /repos/search like with GIN is the only real option here.

Do you want the hub.datalad.org repos to only be queries/fetched/updated once a week like with GIN?

we might want to add some kind of "filtering" based on sensing on whether they have e.g. git-annex branch.

Do you want this or not?

[mih]

I cannot say what is wanted or not. But not every repository will be a dataset.

In any case, the results will be dominated by thousands of HCP (sub)datasets.

[yarikoptic]

@jwodder

@yarikoptic It looks like listing all the repos via /repos/search like with GIN is the only real option here.

Do you want the hub.datalad.org repos to only be queries/fetched/updated once a week like with GIN?

Yes

we might want to add some kind of "filtering" based on sensing on whether they have e.g. git-annex branch.

Do you want this or not?

how do you see that potentially be done? I do not think following up with a separate query per repo would be scalable.

@mih

I cannot say what is wanted or not. But not every repository will be a dataset.

In any case, the results will be dominated by thousands of HCP (sub)datasets.

That's ok -- at this stage we already index non-datasets and non git-annex repos. But then at https://registry.datalad.org/overview/ stage we have ability to filter as desired and get stats on overall git-annex usage.

In any case, the results will be dominated by thousands of HCP (sub)datasets.

That's ok . We already have thousands from https://github.com/dandizarrs and alike. They do get groupped by the "organization" so should be ok, e.g.

[jwodder]

@yarikoptic

how do you see that potentially be done? I do not think following up with a separate query per repo would be scalable.

A query per repo is kind of the only way to do it. One other possibility would be to only make this query when each repository is first added to the database, on the assumption that the result won't change, but we'd still end up doing the query for every old, non-git-annex repository every time the update is run (not to mention that we'd need to query everything on the first run).

[yarikoptic]

@jwodder Greedy me doesn't want to miss some repo which becomes git-annex'ified, so for now let's just get them all as we do for GIN. Longer term - I think we would need to look into seeing API extended to allow for desired queries/limits.

[jwodder]

@yarikoptic I'm implementing hub.datalad.org support using the same code as for GIN, but the GIN code already makes a HEAD request to /repos/{repo}/raw/{defbranch}/.datalad/config for each repo returned, and for some reason, the hub.datalad.org API doesn't accept HEAD requests to this endpoint. Should I change the method to GET, skip the check only for hub.datalad.org, or do something else?

[yarikoptic]

indeed, seems to be not supported by forgejo for some reason... let's do GET but also with Range so we do not bother fetching much either

❯ curl -I -G -H "Authorization: token ${DLHUB_RO_TOKEN}" -H "Range: bytes=0-9" https://hub.datalad.org/api/v1/repos/www/datalad-blog/raw/main/.datalad/config
HTTP/2 405 
allow: GET
alt-svc: h3=":443"; ma=2592000
cache-control: max-age=0, private, must-revalidate, no-transform
date: Wed, 19 Feb 2025 13:59:03 GMT
server: Caddy
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN

❯ curl -G -H "Authorization: token ${DLHUB_RO_TOKEN}" -H "Range: bytes=0-9" https://hub.datalad.org/api/v1/repos/www/datalad-blog/raw/main/.datalad/config
[datalad "%                                                                                                   

did not find an existing issue, so filed a new one https://codeberg.org/forgejo/forgejo/issues/6992

[matrss]

FWIW you can also use the non-API raw endpoints with a HEAD request, i.e.:

curl --head https://hub.datalad.org/www/datalad-blog/raw/branch/main/.datalad/config

[jwodder]

@matrss Are you a developer of hub.datalad.org? It seems that all requests to https://hub.datalad.org/api/v1/repos/jsheunis/annotate-trr379-jsh/raw/main/.datalad/config get a 500 error.

[jwodder]

@yarikoptic Problem: As I stated in the previous comment, one of the requests for .datalad/config for a specific repo on hub.datalad.org is consistently returning 500. While @matrss's URL succeeds for this repository, it cannot be used with gin.g-node.org. What should be done about this?

[matrss]

@matrss Are you a developer of hub.datalad.org? It seems that all requests to https://hub.datalad.org/api/v1/repos/jsheunis/annotate-trr379-jsh/raw/main/.datalad/config get a 500 error.

I am the current maintainer of forgejo-aneksajo, which should mostly be what is running on hub.datalad.org.

I can reproduce that 500 error on codeberg as well. The cause is that the repository is completely empty, so some error is expected anyway. As soon as the repository is initialized with anything that endpoint will start returning a 404 (as long as the .datalad/config file doesn't yet exist).

I agree that a 500 is not really a fitting status code to return here, but maybe you can just treat anything other than 200 as failure and "not a DataLad dataset"?

[yarikoptic]

FTR: for the gin -- we do get 404

❯ curl -I -G -H "Authorization: token ${GIN_TOKEN}" https://gin.g-node.org/api/v1//repos/yarikoptic/testempty/raw/lkajsdf/.datalad/configs
HTTP/1.1 404 Not Found
Date: Wed, 19 Feb 2025 16:47:51 GMT
Server: Apache/2.4.62 (Debian)

so let's treat 500 for forgejo as an indicator of absence for now. @matrss - would you be so kind to submit a fix? ;-)

[jwodder]

@yarikoptic https://atris.fz-juelich.de/ and https://hub.trr379.de/ don't have sign-up options, so I can't get API tokens for them. How should we handle this?

[yarikoptic]

Upon some behind the curtain discussion - let's skip those for now.

[matrss]

@yarikoptic https://atris.fz-juelich.de/ and https://hub.trr379.de/ don't have sign-up options, so I can't get API tokens for them. How should we handle this?

I might be missing something obvious, but why do you even need an API token?

As the admin of ATRIS I could give you a bot account for this purpose, but I think all the relevant API endpoints should not require authentication to index public repositories. Using an account for these requests would also add repositories with "internal" visibility to the results, which might actually be undesirable to index as they are at least not fully public.

[yarikoptic]

Words of wisdom @matrss and thank you for the offer -- frankly, I thought that API access is either entirely forbidden or rate limited (like github does) for non-authenticated access. But indeed, API and /search endpoint at least is available without authentication. May be @jwodder could shine more light.

And thank you for chiming in, so it would be ok with you if we query/index public repos on ATRIS?

[matrss]

And thank you for chiming in, so it would be ok with you if we query/index public repos on ATRIS?

Yes!