Improve Solid compatibility in WebID implementation

[happybeing]

I've been looking into the experimental WebID implementation to understand how I can use this to demonstrate Solid applications using the SAFE Browser WebID drop down. I think I can do this without changes, but not in the typical way of Solid, where pod storage is present at the base URI of the WebID, and both storage and profile are accessed via paths based on the same subdomain and host.

For example, for the WebID https://thewebalyst.solid.community/profile/card#me I have a GUI at https://thewebalyst.solid.community/, storage at folders beneath this (e.g. https://thewebalyst.solid.community/public, and my WebID profile document at https://thewebalyst.solid.community/profile/card.

The experimental APIs cannot replicate this as things stand, so this issue is a request for these to follow Solid conventions (and so allow the above), unless there are good reasons not to.

At the moment I think the best I can do is to require Solid apps to use storage at a separate URI, which I think will work, but will not make as good a demo, or case for people to try Solid apps on SAFE. So ideally I'd like to try and replicate the above.

I've explained below, what I think the current implementation does, and the options for changing the implementation. I think the options are a) long term: include www service storage in a way that allows default APIs to follow the Solid conventions, and b) short term: fork the WebID Manager to create a Solid WebID manager that I think will allow the behaviour, while bypassing the RDF serialisation we have right now.

I'd appreciate feedback on the following, advice and suggestions for alternatives etc, before I embark on the option b) above

1) Current Experimental WebID Implementation

If the WebID is present, it will have as its service container a Mutable Data containing RDF, which is incompatible with the SAFE www service. Instead:

• webFetch() on any URI beginning with the WebID (e.g. safe://me.happybeing) returns the WebID profile document.
• webFetch() on the public name (e.g. safe://happybeing) returns "Requested service is not found." and there is no mechanism to add storage (e.g. for a www service as an NFS container) based on a WebID name. This feature could be added by extending the experimental API in webFetch(), fetchHelper() etc to support additional services on different subNames, but still not on the WebID URI itself (i.e. safe://me.happybeing). In Solid, this is generally assumed to be the default (i.e. storage would normally be provided at paths based on the WebID, such as safe://me.happybeing/public).

Long Term (option A)

To support storage based on the WebID URI itself would need the current implementation to change so that the storage for a web service and for the WebID profile use a common approach. This would mean that the profile document would appear within storage located at the URI of the hostname used in the WebID, as is the convention with Solid (e.g. safe://me.happybeing/card for the profile, and safe://me.happybeing/card#me for the WebID). I think it would be helpful for SAFE RDF APIs and default UI (e.g. for WebID creation) to follow Solid conventions such as this unless there are good reasons not to.

2) Bypassing the Experimental WebID Implementation

If there were both WebID and normal public name entry using the same name (e.g. happybeing), the public name entry and service will be used, and the WebID profile will not be accessible.

Short Term (option B)

So I propose making a WebID is changed to create the RDF entries as now and create a regular public name and 'www' service on the same name, then store the WebID profile as an NFS Turtle file in the www container. Then I think this implementation will be able to follow Solid conventions. If so this would be a useful short term way to demonstrate maximimum compatibility with Solid UX and applications, pending changes to allow this in the exising WebID/RDF serialisation APIs (option A).

Note: I've assumed that the SAFE Browser WebID drop down feature would not be broken by the presence of both a regular public name and a WebID using the same name, and have not looked into that. I think that provided we keep both copies of the profile in step, there should not be an issue (so ok for demos and Solid developers wishing to evaluate SAFE). Any comment on this would be useful.

To achieve this interim solution, all that I think is needed is a fork of the WebID manager and calling it a Solid WebID manager to reduce confusion.

I'm not entirely sure option B) will work, so would appreciate some eyes on the idea before I test it out, and as noted welcome any comments and feedback, and corrections.

[bochaco]

The way we imagined back then when we started with the PoC of WebIDs was that:

• A WebID profile doc is an RDF that, with the new data types, would be presumably stored on PublishedAppendOnly data type
• Then, like any other document/file which needs to be organised in a folders/directories/paths hierarchy, it needs to be linked from a FilesContainer, which would replace the NFS containers by introducing RDF in its data representation
• Once we have the FilesContainer which maps paths to files/docs, we link it from the subname within a publicname, i.e. in a ResolvableMap (as defined in RFC 52)

Thus, if you try to fetch safe://www.publicname/profile/card#me we resolve it with:

• Get the ResolvableMap located at sha3('publicname') which contains the mapping for all subnames of 'publicname'
• Resolve www by looking up in the ResolvebleMap, and fetch the linked FilesContainer
• Resolve the path /profile/card from the FilesContainer and fetch the linked WebID document (i.e. the PublishedAppendOnly doc)
• Resolve #me from the WebID profile doc to get the specific WebID

[happybeing]

Thanks @bochaco that's very helpful. Assuming those changes are months away, in the interim do you think option b) will fly? I'm not planning to implement that initially, but it would be good to have it sanity checked while I do the other approach (NFS storage on a separate NFS URI).

EDIT: can you explain why you are creating entries in both _public and _publicNames? There are a few implementation details I haven't grasped, but that one is niggling me! 😄

[bochaco]

If I understand correctly, you plan to serialise a WebID and store it as an ImmutableData file using the NFS API, and map it to a subname/service name?

If so, yes I'd consider it a valid path, moreover, I'd consider it not only in the short term, it's just about the use case. If you think about it, the only difference there with what I described (apart from the new data types and RDF representation of this data) is that your WebID cannot be edited that's all, well, you can by uploading a new ImmD file and update the NFS entry, while in my description you'd edit the WebID profile doc itself and therefore wou'd be able to version it and so for. What I mean is that in the future different tools may implement different ways according to what they try to cover, if I want to be able to have versions on my WebID profile doc I won't use this approach you are going for now, but other may prefer that for XYZ reason...?...

The _publicNames is your private inventory of public names you own, so you can find them even if you forgot you created them.
Regarding the _public container, if you refer to the WebIDs linked from there, we needed to easily be able to get the list of WebIDs owned by the account, so going thru the _publicNames, then looking up in the public name all services that were effectively WebIDs and not other things like websites, to then come up with a list of WebIDs was too complex and time consuming, so we thought that we eventually may need a private container to be an inventory of WebIDs, so analogous to the _publicNames but for WebIDs profile docs, e.g. _webids, but since we didn't and don't have that container we used the _public as they (WebIDs) were meant to be public. In the future, as we propose in the CLI design doc, users should be able to have any type of container linked from the account, so they/apps could create a container to keep the inventory of WebIDs.

[bochaco]

I think I'm starting to push for calling these containers Inventory containers :)

[happybeing]

@bochaco Thanks for the explanations, helps a lot. BTW my short term plan is only meant as a way to get the feature working for demos and evaluation. I don't see a reason to have multiple mechanisms at this time, so once the RDF APIs support storage as you anticipate I'd go back to using that.

Do you think that my creating a regular public name as well as the existing RDF API WebID would still allow the WebID browser drop down to work properly? From the sound of it so long as I am not messing with the entries in _public/webId that should be happy, and users can select and apps can access, an active WebID as now. My solid-auth-client fork will just take that and pass it to the Solid app in the expected form. Thanks for your time! 😄

[bochaco]

I think you may not need to do much really, if you take a WebID doc, serialise it with ...Turtle? I think you like it :D , then you store that turtle file as a text file using the NFS API with the path /profile/card, then create a public name and subname, let's say www.veryhappy for that NFS container, you should be able to webFetch it with safe://veryhappy/profile/card#me, what you get is the Turtle text file back, from there on it's up to the app what it needs to be done with that document, including making use of #me to read/consider that particular WebID from the profile doc. I haven't tried this, but I think it should work if you use what we have in alpha2 and use the webFetch function from safe-app-nodejs or DOM API

[happybeing]

Yeh, I was already happy with that part. My question relates to not wanting to break the Browser WebID UI or the WebID API by adding in a public name entry alongside the one for the WebID.

So my forked WebID manager will do what the WebID manager does now, and create a public name identical to the name of the WebID (as well as adding the duplicate profile as you just described).

I can't see a problem, but was asking if you think there's anything that might be broken by having WebID RDF entries and regular public name entry in _publicNames for the same name. Since you explained what the _public/webId entries are used for I figure I'll be OK as I'm not messing with those but wanted to run that by you too.

Don't worry about digging further, I will do a test when I get to this, but will start with separate storage and implement this later.

[bochaco]

ahh I'm sorry, we tried to make the code to support entries in _publicNames which are not RDF, this is how we assured compatibility with what WHM creates, so it should be fine, in fact I think what you'd be doing is the same as WHM does, so all good.

I also was thinking you could even use the WHM to pusblish a WebID in this form, create a text file with the Turtle and upload it as part of a website with WHM, if you then try to fetch it with the browser it should offer you to download the Turtle file (it won't recognise it as Turtle but just as an octet stream).

[happybeing]

My aim is to demo using the SAFE Browser WebID UI, so I will have to maintain both side by side, which is why the question arose... might that break anything?!

[happybeing]

I've coded the support in a fork of web-id-manager-js to add storage at a separate publicName related to the WebID host. So for example, on creating a WebID at safe://me.happybeing#me it also creates a public name with www storage at safe://ldp.files-happybeing. This storage be accessible either by browser (www) as normal, or by a Solid app using an 'ldp' service.

I see that to publicise the storage URI (e.g. using space:storage) in the WebID profile, I have two options. One is to use the emulateAs('RDF') API to add the triple. This is fine, except it doesn't make this available as part of the WebID object. So to support that I'll have to also add support for a 'storage' property to the WebID profile in https://docs.maidsafe.net/safe_app_nodejs/api_emulations_web_id.js.html

If I add support for 'storage' in safe_app_nodejs/api_emulations_web_id.js it means a custom build of the browser is needed for any app that wants to access 'storage' via the WebID object. For my demo I assume I can avoid the need for a custom browser by going directly to/from the RDF emulation, but this has highlighted a problem for me with the current API: that while a WebID profile should be able to have arbitrary data included in it, this is hard once you want to manipulate anything outside the simple WebID json profile based API. I guess you are aware of this!

I'm wondering how the approach might / might not change going forward, in order to provide a simple API without limiting it to a small subset of what might be held in a WebID profile. Is it always going to be either a fixed set of properties on a JS profile object, or go directly to RDF? Or do you anticipate a more flexible but still simpler API (than the RDF emulation)?

[happybeing]

UPDATE and a question: I have a fork of WebID manager which creates NFS storage for a WebID and inserts a space:storage triple into the SAFE WebID indicating the storage URI.

However, I think I've hit a limitation with the current RDF implementation and requested an enhancement to how services are implemented, see issue #377.