Content of Turtle and RDFa documents should be wholly and entirely preserved

[TallTed]

Originally discussed by @RubenVerborgh, @kjetilk, and @TallTed in #301 (comment) and preceding

[@TallTed] I am rather weary of explaining this over and over, and hearing back, "we don't care about your idea of what data is."

[@RubenVerborgh] I do care about your idea; but we need some pointers to back that up.
And insights into the consequences of adopting that viewpoint versus another; I see many thing we'd not be able to do, not really gains. I don't know any sources that consider RDF whitespace and comments to be data.

If they need to be data, then the MIME type should not be RDF, but rather text/plain or similar (or maybe even better application/octet-stream to avoid encoding issues), and then data will be preserved.

[@TallTed] then non-RDF contents of Turtle and RDFa files MUST be preserved and these files MUST be treated as ldp:NonRDFSource.

[@RubenVerborgh] A citation would be helpful.

My understanding of the matter is that a content type determines what transformations are allowed with preservation of semantics. This is not unique to RDF at all. Minified JSON carries the same meaning as the unminified version.

(@TallTed Should you want to continue the discussion, could you please open another issue as per @kjetilk's suggestion?)

[TallTed]

There is no citation for you, @RubenVerborgh, because those MUSTs are not taken from RFC, REC, or the like. They do, however, flow from the understanding that I believe was commonly shared among the LDP WG when writing the LDP REC, upon which Solid at least claimed to be based at one time, even if Solid is now dis-claiming that basis.

One of Solid's early claims, if not promises, was to be a filesystem-ish datastore, which could be backed by an RDF store and could support SPARQL over the stored data, but both SPARQL and RDF features were considered a heavy-lift, so these were not promised.

As an author of RDFa and Turtle documents, I consider the HTML content in the former and the whitespace and comments in the latter to be important -- else I would not have created RDFa at all, nor added whitespace and comments to the latter to make human processing of these files easier.

In other words, if I did not care about the non-RDF content of these documents, I would have chosen a more "pure" RDF media type that did not support comments and/or author-applied whitespace -- which might include JSON-LD, which certainly does not support comments, and does not promise to maintain whitespace.

Still, even with more "pure" RDF documents, I would not expect them to be broken down into their component RDF and only that saved in an RDF store without clear warning and user choice. I would generally expect the documents to be preserved as documents, complete with all their metadata (creation date in particular, but also modification date, and any other metadata supported by and transported from their origin to the Solid store).

To some of your specific comments...

I don't know any sources that consider RDF whitespace and comments to be data.

There are no "RDF whitespace and comments". There are syntactically valid, and presentationally and contextually valuable, whitespace and comments in Turtle documents. There is syntactically valid, presentationally and contextually valuable, content (a/k/a data) in RDFa documents.

My understanding of the matter is that a content type determines what transformations are allowed with preservation of semantics. This is not unique to RDF at all. Minified JSON carries the same meaning as the unminified version.

I, and I am quite sure others, would not be happy to have some of their data discarded without warning because you, and possibly others specifying and coding Solid and related tools, decided that their Turtle- or RDFa-contained data was not really data.

To my knowledge, there is nothing about the IANA-maintained media type lists which says "this part of this media type's content is real data and SHOULD/MUST be retained, but that part of this media type's content is not 'real' data and MAY be discarded."

Minified and unminified JSON (and JSON-LD) are trivially and losslessly transformable into each other. There are no comments to be retained or lost, because JSON doesn't support comments (as of today, that is; I believe an upcoming JSON version, already in progress, will add support for inline comments). Inline newlines are only supported as \n\r and the like; there is no human-friendly indentation available except that which is likewise machine-friendly. These are in marked contrast to Turtle and RDFa which do support human-friendly content that is distinct from, yet intermingled with, their machine-friendly content.

Again, I have no RFC, REC, nor other "standard" or "specification" citation to offer, in significant part because the standard-setting bodies working on relevant specs in which I've participated agreed with my assessment above -- that non-RDF content of RDF-encoding media types is just as important as the RDF content, and all should be preserved -- and acted and discussed other things with the preceding as foundational and self-evident. Had there been any disagreement, I would have made sure that explicit statements to that effect were included!

[kjetilk]

I understand the desire, but I do not understand that there has been a clear historical path here.

LDP defines:

Linked Data Platform RDF Source (LDP-RS) as "An LDPR whose state is fully represented in RDF, corresponding to an RDF graph." which in very clear terms do not include whitespace, comments, etc.

Then,

Linked Data Platform Non-RDF Source (LDP-NR) as "An LDPR whose state is not represented in RDF. For example, these can be binary or text documents that do not have useful RDF representations. "

The class of resources that you want are clearly not included in either of these classes, this is a class of resources that have a useful RDF representation, but where the representation does include more than RDF that is significant. If that understanding was present in the LDP WG, I don't think it was in any way adequately expressed in normative statements.

I believe that we can satisfy this by introducing such a class, and perhaps we should (I certainly can feel the pain around HTML+RDFa, and I'm sure @csarven feels it even more). I just don't think that can be a requirement right now, even if NSS could support that.

[TallTed]

For the umpteenth time: I was an active participant in the LDP WG.

I am reasonably certain that I understand what we meant, frustrated that we failed to communicate that, and more frustrated that my clear statements of our intended meaning (which I believe have yet to be disputed by anyone else who participated in the LDP WG, leaving me with no feeling of having misunderstood our intention) to clarify others' misunderstandings and misinterpretations are discounted and/or ignored.

Our simple error here lies in having left out one word from the LDP-NR definition: "fully".

We did not intend to have three types of LDPR, with two identified and defined explicitly as noted above and one identified and defined only implicitly by the gap between those two.

We intended to have two types of LDPR, as explicitly identified above, i.e., LDP-RS which are "fully represented in RDF", and LDP-NR which are "not [fully] represented in RDF". Nothing in the definition above says that an LDP-NR cannot include RDF as part of its payload, only that RDF is not all of its payload.

[csarven]

We've been through this material and discussion several times. I've been content to assume that Ted is sharing knowledge of LDP in good faith for some time now. Unless the editors or authors of the LDP spec or LDP WG members can clarify the language, I suggest we take that as is. Debating whether the language in LDP is meaningful or intuitive against Ted's explanation of what's intended is not particularly useful at this point. (I've been down that road. We don't all have to.)

Having said that, the Solid Protocol - or at least some of the original servers and clients - neither assumed or required LDP compliance. (I know because I wrote a client that expected the servers to implement LDP but they didn't actually deliver despite the fuzzy claims.) NonRDFSource and RDFSource was not something adopted or deemed necessary for Solid. It seemed to create more concerns than actually being practical. From my perspective, this whole thing turned into moving the Solid Protocol forward without bringing the whole LDP baggage (YMMV). But we are not completely free since we do expect containers in Solid to behave like LDP(B)C. Perhaps we just need solid:Resource/Container (semi-serious suggestion).

[TallTed]

In early-ish days, Solid tied itself loosely to LDP, which was supposed to be a good thing because LDP servers were supposed to require minimal adjustment to also be Solid servers.

I am content to let Solid break from that loose binding IFF it doesn't lead to silently broken expectations of document preservation.

In my world, when I upload a document -- whether that be Turtle, HTML+RDFa, JPG, PNG, CERT, or otherwise; and whether or not the store to which I'm loading it knows how to extract information from that document -- to a document store of any kind, I expect that document to be retrievable in the same condition, with the same content, as when it was uploaded unless it was edited/changed by some process I explicitly either approved or initiated.

That includes when I write a Turtle or HTML+RDFa document to a Solid pod.

[TallTed]

"First, do no harm." Or more aptly, "First, lose no user data."

Also, there's the whole "distribution of effort" credo (which I can never bring to mind by the name others use for it) that says users should have to think the least, in order, after deployers, after programmers, after specifiers...

We (OpenLink) use a lot of Turtle in our line of business applications (like this document), among other things. If you take a look there, you'll find a fair amount of comment and whitespace content included to aid human comprehension and future text-editor-based revision.

# A rather excruciatingly detailed self-describing Turtle snippet
###
# First some PREFIX definitions, all right-aligned on the `:` ending the prefix
# The first one defines the most important prefix, that refers to this document, ":"
###
PREFIX      :   <#>
###
# Then some prefixes for commonly used ontologies, 
# all alpha-sorted ignoring whitespace, to avoid duplication 
# and to ease human parsing of this sample dataile
###
PREFIX  foaf:   <http://xmlns.com/foaf/0.1/>
PREFIX  rdfs:   <http://www.w3.org/2000/01/rdf-schema#>

###
# Then some RDF data
###
:this     a                  rdfs:Resource ;
          rdfs:description   """This text is to demonstrate the 
                                utility of inline whitespace
                                within Turtle data.""" ;
          rdfs:comment       "just a bit of text" .
###
# And then the document ends.
###

The more unfamiliar the realm of discourse, the more important such refinements are.

Now, users could describe one Turtle file with another, and annotate the first with line-number-based identifiers, instead of inline comments, but that puts a lot of work on the Solid user, which is not required by any other document repository ... but once your Solid server that's refusing to preserve Turtle documents in toto ingests the content of those two files, there is no guarantee that the line-number-based descriptions from the second will remain accurate for any Turtle file that is eventually output with the same RDF graph as the first, because there's no ORDER required by SPARQL, etc.

[csarven]

I'd suspect the WG captured some of the background in the mailing list, meeting minutes or the W3C issue tracker. I suppose we weren't compelled enough to dig into that - but that shouldn't stop anyone from doing that now.

LDP's constraints re "fully" is (unnecessary complex) plumbing either way.

If we need a class to distinguish RDF stuff from non-RDF stuff, we can use solid:RDFDocument or solid:RDFSource (based off rdf11-concepts) :) The rest is just media types, multiple representations, graph comparisons.

---

Sure, that "loss of quality" is not about RDF and the request is specifically for a representation that's a lower quality (i.e., usually in one of the concrete RDF syntaxes other than RDFa). Shrug.

If we are not talking about RDF graph comparison, "loss of quality" can be expected when going from any concrete RDF syntax to another.

[kjetilk]

@TallTed I certainly acknowledge your contribution to LDP and Solid. It is just that expectations aren't much what implementors base their code on.

The difference, from an implementation standpoint given those assumptions is large, for RDF sources, it is clearly legitimate to persist only the RDF graph and so you use technologies available for that like a quad store. and serialize when people make requests. For non-RDF content, it is clearly legitimate to not persist the RDF graph and so, it creates a divide. For a third class, you'd have to do both, and so, the specification needs to be abundantly clear about it.

I am already quite convinced that we need it, I'm not as hard to convince as @RubenVerborgh , but still, there is a piece of work that needs to be done.

[TallTed]

@RubenVerborgh @kjetilk -- Am I not, as a Solid user, able to store arbitrary documents in my pod, as well as application-generated data files? I know, everybody's first thoughts are about pictures and movies, but textual documents are also among the commonly shared sorts of thing, and there's no reason why I, as an early Solid adopter, might not hand-edit some Turtle about those pictures and movies, or about some stuff that's not now (and might soon or might never be) in my pod.... What you're telling me here is that I must not be a tinkerer, I must not hand-edit my Turtle to please me, because only the app authors are allowed to do this.

[kjetilk]

@TallTed you most certainly are able to store anything. As for tinkering with files, you can do that too with a Solid server that supports that, like NSS and CSS with certain backends. But you'd also have to be aware that if you're not editing through means provided by the protocol, it is going to be things that are hard, like you need to make sure that containers are updated if you create a file on the file system, that etags and last-modified times are updated, that you protect the data from unauthorized access, etc. It doesn't exclude that kind of assumptions, but it allows servers to make an assumption that it persists the RDF graph.

[kidehen]

@RubenVerborgh @kjetilk -- Am I not, as a Solid user, able to store arbitrary documents in my pod, as well as application-generated data files?

In text/plain.

When did that become the case? You've always been able to store "text/turtle" by hand to a Solid Pod.

A Solid Pod provides a filesystem-like experience to Agents (users or machines).

I don't understand your response here, please clarify.

/cc @timbl @TallTed

[TallTed]

@RubenVerborgh -- I would be delighted to not need to keep harping on this issue, but where comments by you and others in other issues demonstrate the validity of the assertions I've made here, which have yet to be accepted here as valid by you and others, I think it is valid to say so there.

Your own sample JSON data incorporated comments which made that sample invalid as JSON, and when I flagged that, you noted that you could have used Turtle where your inline comments would have been valid. Are you now asserting that those comments were not important enough to retain? If so, I have to wonder why they were important enough to include in the first place ... and further, how other readers of your JSON snippet were meant to fully comprehend it, as the content of those comments was not replicated outside of the JSON snippet.

[TallTed]

You know very well that everyone on that page knows JSON does not do comments, and Turtle does.

You are making presumptions about potential readers with which I disagree. There is no certainty that "everyone on that page" knows any particular thing, including the disparate (lack of) support for comments in JSON and Turtle.

It is my feeling that you want to exile any comments about full preservation of documents which you don't think need to be fully preserved, and gloss over even your own comments elsewhere which support my position and undercut yours.

I would have thought it clear that I am not trying to sneak anything in anywhere. Rather, I am speaking in full view, where it seems relevant to me to do so.

I will also thank you not to deride my comments as mere "vent[ing] about #342", which suggests that there is no merit whatsoever to my position, which you have justified simply by the fact that my position is not yours.

If Solid is going to go forward without promising to preserve the full content of documents which Solid has decided are not worthy of full preservation, then I believe Solid will die a deservedly painful death, and has the potential to ruin many users' days along the way, as they discover the loss of their data. I would prefer that neither of these fates come to pass, and rather that Solid preserve the content of my, and their, documents.

[TallTed]

The most straightforward step is to warn the user when document content -- whether it's HTML within RDFa, or comments and/or whitespace within Turtle, or something else -- is going to be lost through that PATCH.

I believe there's been general agreement that HTML+RDFa documents should be preserved, and so they might not be PATCHable. I think the same logic should be applied to any other document formats a/k/a media types that (may) include both "in-band" RDF and "out-of-band" non-RDF, including but not limited to Turtle.

The requirement to update resources via PATCH could be adjusted to require the ability to update only those resources which do not lose other content (which may may not be "near" the intended update) when so updated. In other words, PATCH might be required only for pure RDF content, such as JSON-LD documents, which cannot contain out-of-band, non-RDF data. Alternatively, PATCH might be required only for RDF which is stored in a triple/quad store, whether it arrived there by extraction from some impure, partially-RDF document or by ingestion of some pure, wholly-RDF document; the originating document could be retained, unchanged, while the RDF in the triple/quad store gets PATCHed.

My proposed requirement is not to "preserve comments" but to "preserve all document content other than that which is explicitly and intentionally and informedly altered or deleted, with user consent, by user action" which might take the form of a confirmation dialog regarding a Turtle document, e.g., "Your PATCH of these statements within the document will cause the following to be lost: {possibly line numbered, blah blah blah, etc.} Proceed? Y/N"