Expose linked projects/files in API

[dschreij]

We are currently bumping into this limitation in the OSF Explorer of OpenSesame: open-cogsci/QOpenScienceFramework#10.

Specifically, we would like to provide an option for users to view public projects of which they are not a collaborator. In the OSF web interface you can create links to other projects, but they do not show up in the list of nodes that is provided by the OSF API. Is there a way to do this?

[brianjgeiger]

Hello! Try https://api.osf.io/v2/nodes/ for access to all nodes that the currently authenticated user can access, public and private.

[smathot]

Hi Brian,

Daniel's question is basically a polite feature request.

• Linked OSF projects show up on the OSF website as storage providers, just like for example a linked GitHub repository. This makes sense.
• But Waterbutler does not expose linked OSF projects as storage providers (and probably not at all). This doesn't make sense.

So our feature request is: Could Waterbutler be made to expose linked OSF projects as (read-only) storage providers, in the same way that other storage providers are exposed as well. That would be in line with Waterbutler's goal of providing a fully transparent API to multiple backends, right?

That's my understanding. (Correct @dschreij ?)

Cheers,
Sebastiaan

[dschreij]

Yes, that's what I actually meant. Sorry if I was a bit unclear. The endpoint which can retrieve nodes that are accessible by the user is useful, but only if you actually have the id of the node you want to get the information of to begin with.

The osf-explorer in OpenSesame starts with requesting a list of nodes owned by the user and traverses the children of these nodes recursively to build up the file tree. The children of a node are retrieved by GETting the link provided in its data:

...
  "children": {
                    "links": {
                        "related": {
                            "href": "https://api.osf.io/v2/nodes/rxpkp/children/",
                            "meta": {}
                        }
                    }
                },
...

If linked nodes (projects, files, folders, etc.) are not part of the data that is returned by this link, then the osf-explorer will not know of its existence and will not add it to the tree. So the only way for getting linked nodes to show in the file tree is if linked nodes are part of the list of child nodes that is provided by the above API endpoint.

[brianjgeiger]

We don't return linked nodes with children because linked_nodes have different rules and you can link to other things aside from nodes. One of the different rules is that the parent-child relation is one-to-many, but linked nodes are many-to-many with no constraints on the relationship, which always causes people to go into infinite loops when they're trying to use the parent-child relationship as a tree. Especially if someone links to the parent node, other ancestor/descendant node, or to itself.

However, the information is available; you can grab it from the linked_nodes and linked_registrations relationships. As we add more link types (preprints seems a likely candidate, for example), then those will likely be useful as well. Does that help?

[dschreij]

That being said, the endpoint @brianjgeiger provided does help in another challenge we faced: getting an experiment file from the API by url (open-cogsci/QOpenScienceFramework#11).

However, it seems that the node IDs shown on the web interface on osf.io, do not correspond with the node IDs used by Waterbutler. if I have a public file I want to link to, for example https://osf.io/6kw2p/, I assume the ID of the node representing this file is 6kw2p, but when I try to get its information with

GET https://api.osf.io/v2/nodes/6kw2p

I get the response:

{
  "errors": [
    {
      "detail": "Not found."
    }
  ]
}

so it seems the osf.io IDs do not map one-on-one with the api IDs, am I correct?

[brianjgeiger]

GUIDS can point to different kinds of things. For that, I expect https://api.osf.io/v2/files/6ke2p/ is what you're looking for. If you have a GUID that you don't know what it goes to, https://api.osf.io/v2/guids/6ke2p/ will redirect you to the actual endpoint.

[dschreij]

Thanks Brian, so we need to parse the GUID from the URL (which should not be that problematic and quite robust since the osf.io urls that contain them have a fixed pattern), and then use that (GU)ID with the endpoints you just mentioned. Sounds doable.

[brianjgeiger]

Yes. Ideally you're meant to use the OSF API as the entry point for everything (including waterbutler access), which should give relationships for everything and keep you from having to construct a URL, but in the case where all you have is a guid, for example, that pattern is a good one. If you hit any other snags, let me know.

[dschreij]

I've been looking into this matter a bit more thoroughly and see that linked projects are provided by the API. However, the same information seems to be provided twice with a different key/identifier, making things ambiguous. If I request the data of a certain project, which links to another project, I see the following entries in the API response:

    ...
    "node_links": {
        "links": {
          "related": {
            "href": "https://api.osf.io/v2/nodes/hz9xv/node_links/",
            "meta": {}
          }
        }
      },
      "linked_nodes": {
        "links": {
          "self": {
            "href": "https://api.osf.io/v2/nodes/hz9xv/relationships/linked_nodes/",
            "meta": {}
          },
          "related": {
            "href": "https://api.osf.io/v2/nodes/hz9xv/linked_nodes/",
            "meta": {}
          }
        }
      },
     ...

Both of these links (either https://api.osf.io/v2/nodes/hz9xv/node_links/ or https://api.osf.io/v2/nodes/hz9xv/linked_nodes/) appear to offer the same information at first sight. Is there a difference between them, or is one deprecated but still present for backward compatibility reasons? Which one do you advice to use?

[brianjgeiger]

Use linked nodes. Node links are deprecated and harder to use. That should be much more obvious in our docs and in the API response, so my apologies there. We will get that updated.

[dschreij]

I figured as much and went with linked_nodes already. Thanks very much! One related question: the access permissions are only listed at the project level at the moment. This makes it difficult to determine the access rights of individual files and folders inside a project. To do so, you need to check the parents of each file/folder, until you get to the project node that contains this information. Would it be possible to add this permission information to each child node of a project?

I am now copying this information into the tree representation of the OSF information that I maintain client-side, but this seems like a wrong (and error-prone) approach to me.

[brianjgeiger]

Hmm. The permissions are stored on the node, and files / folders don't have their own permissions as such. Although we will at some point have read-only file sources. We will look into that. Could you file a separate issue for it and reference this conversation?

[dschreij]

This issue can be closed, as the API already provides information about linked projects.