How to do links with content negotiation: Adding "types" to "link"

[joanma747]

Lets imagine that we would like to expose that we can retrieve a map in 5 formats:
"image/jpeg, image/png, image/gif, image/tiff"

If the web API have to support content negotiation ("Accept:" in the headers), how can we express this in a map description document link?.
At this moment it is unclear to me. If you list all possibilities how the client will know how to do content negociation?

Possibility A (the same href):

"links": [
       {
          "href": "http://data.example.com/collections/buildings/map/brown",
          "rel": "subset",
          "type": "image/png",
        },{
          "href": "http://data.example.com/collections/buildings/map/brown",
          "rel": "subset",
          "type": "image/jpg",
        },{
          "href": "http://data.example.com/collections/buildings/map/brown",
          "rel": "subset",
          "type": "image/gif",
        },{
          "href": "http://data.example.com/collections/buildings/map/brown",
          "rel": "subset",
          "type": "image/tiff",
        }
]

Possibility B (as suggested by the Maps Sprint to make it more compact:

"links": [
       {
          "href": "http://data.example.com/collections/buildings/map/brown",
          "rel": "subset",
          "type": "image/png",
          "types": "image/png, image/jpeg, image/tiff"
        }
]

Possibility C (that is fine but wher is the content negociation in this one?:

"links": [
       {
          "href": "http://data.example.com/collections/buildings/map/brown?f=png",
          "rel": "subset",
          "type": "image/png",
        },{
          "href": "http://data.example.com/collections/buildings/map/brown?f=jpeg",
          "rel": "subset",
          "type": "image/jpeg",
        },{
          "href": "http://data.example.com/collections/buildings/map/brown?f=gif",
          "rel": "subset",
          "type": "image/gif",
        },{
          "href": "http://data.example.com/collections/buildings/map/brown?f=tiff",
          "rel": "subset",
          "type": "image/tiff",
        }
]

Possibility D

"supportedMapFormats": [
       "image/png", "image/jpeg","image/png","image/tiff"
],
"links": [
       {
          "href": "http://data.example.com/collections/buildings/map/brown",
          "rel": "subset",
          "type": "image/*",
        }
]

In the Maps Sprint we selected the Possibility B. I believe it is a general issue that need clarification from common.

[cportele]

@joanma747 - I think you are misunderstanding the link parameter "type". It is not meant to be a hint which values can be used in the Accept header, it is an optional hint about the content of the response you will get back when dereferencing the link. In that sense, option A with multiple links with the same href/rel will be confusing. Multiple values of type (B) are invalid as is the use of "*" (D).

So, from the options that you list only C is consistent with RFC 8288.

For cases where we think that an API should be explicit and offer "typed links" this is the pattern that should be used, e.g. in the "alternate" links.

In many cases, we don't need to say anything about the type attribute in the specs. The standard HTTP approach is that the client lists the media types it can handle for the resource in the Accept header (including the preferences). If the client receives a 406 or a Content-Type it does not understand, the client knows it cannot handle the resource and will act accordingly. Otherwise it will process the representation that it gets back. So, there is also option E - without a link parameter "type".

[cmheazel]

A section has been added to the API-Common Users Guide (OGC 20-071) to discuss linking and relation types. The content is largely copied from RFC 8288 (with a bit from W3C XLink 1.1). Feel free to propose additional informative content.

[jerstlouis]

@cportele Would there be any proper way to advertise valid types for the client, without having to have a separate link for 100s of different formats supported?
Another related point is how to advertise one representation as the most efficient (e.g. native), or the original one one?
To what point can links be extended? RFC5988 says that links can optionally include target attributes, doesn't it?

[cportele]

@jerstlouis I would say that if the server supports 100s of media types for a resource, I would not include a type attribute in the links and let the client simply use content negotiation to select the best format for the target. If the client supports multiple formats, it can list them with weights in the Accept header.

In general, links can be extended with additional target attributes, but we should be very careful with this and make sure that clients which have no idea about the extensions still work properly.

(BTW: RFC 5988 is replaced by RFC 8288.)

[jerstlouis]

@cportele So that leaves it up entirely to the client to make the first negotiation step in saying what it supports.

It feels a bit odd as it is the opposite of listting multiple links each with their own media type, where the server makes the first move in letting the client knows what media types it supports.

Thanks for clarifying about 8288.

[joanma747]

So in conclusion we can do 2 things from the client side:

• Expose all the formats supported as "separated" links
• Use only one link without the type attribute and entirely rely in content negotiation.

In addition

• OpenAPI can also enumerate the supported formats.
• There are specific conformance classes for formats

The "types" solution is discarded. by this discussion. The problem in OGC API Maps persists: We are trying to avoid to define conformance classes for particular formats so, how to enumerate the list of formats that the server supports for a particular layer?

Actually, giving it a second thought it is not necessary for the server to tell the "world" what the server support. For a particular client it does not matter what the server supports. It is only important what the client supports in the first place. If the server support something "new" that the client does not know, the client will not be able to use it anyway. It is a change in mentality.

If we accept the "client first" mentality, then the server should not expose formats. It is better that the server exposes no media types and provide a generic link. We should forget about the file extension in the URI.

Possible practical solution: we could mix the "client first" with " server first" by exposing just a few "favorite" links with "type=" and a generic URI without "type" for possible "alternatives" and allow the client to use "Accept: " header

(Just an idea: This "change" in mentality could be applied to CRS's too. The client could "request" the "needed" CRS in an "Accept-CRS" header and "Content-CRS" in the response could say which one it returned. If we accept that server should describe itself, then this approach got against what we normally did in the past)

[joanma747]

2020-10-05 telco decision:
We will include the "possible solution" above in the Guide and close the issue.

[pomakis]

At the moment. our server only reports typeless links (with the exception of "self" and "alternate", which intrinsically demand specific content types). E.g.:

  "links": [
    {
      "href": "https://example.com/ogcapi/collections/coastl_1m?f=json&pretty=true",
      "rel": "self",
      "type": "application/json",
      "title": "this collection as JSON"
    },
    {
      "href": "https://example.com/ogcapi/collections/coastl_1m?f=xml&pretty=true",
      "rel": "alternate",
      "type": "application/xml",
      "title": "this collection as XML"
    },
    {
      "href": "https://example.com/ogcapi/collections/coastl_1m?f=html&pretty=true",
      "rel": "alternate",
      "type": "text/html",
      "title": "this collection as HTML"
    },
    {
      "href": "https://example.com/ogcapi/collections/coastl_1m/schema",
      "rel": "describedby",
      "title": "the schema of this collection"
    },
    {
      "href": "https://example.com/ogcapi/collections/coastl_1m/items",
      "rel": "items",
      "title": "the vector features of this collection (accepts query parameters for filtering, etc.)"
    },
    {
      "href": "https://example.com/ogcapi/collections/coastl_1m/styles",
      "rel": "http://www.opengis.net/def/rel/ogc/1.0/styles",
      "title": "the available styles for this collection, with links to style-specific map layers and tiles"
    },
    {
      "href": "https://example.com/ogcapi/collections/coastl_1m/map",
      "rel": "map",
      "title": "a map layer of this collection (accepts query parameters for subsetting, etc.)"
    },
    {
      "href": "https://example.com/ogcapi/collections/coastl_1m/map/tiles",
      "rel": "http://www.opengis.net/def/rel/ogc/1.0/tilesets-map",
      "title": "map tiles of this collection, in the collection's default style"
    },
    {
      "href": "https://example.com/ogcapi/collections/coastl_1m/tiles",
      "rel": "http://www.opengis.net/def/rel/ogc/1.0/tilesets-vector",
      "title": "vector tiles of this collection"
    }
  ]

I figure that this is the appropriate thing to do since section 8.1.7 of "OGC API - Common - Part 1: Core" states:

Components negotiate the encoding format to use through the content negotiation process defined in IETF RFC 7231. Additional content negotiation techniques are allowed, but support is not required of implementations conformant to this Standard.

It also has the pleasant effect of keeping the set of links succinct.

The standard then goes on to state:

Note that any server that supports multiple encodings will have to support a mechanism to mint encoding-specific URIs for resources in order to express links, such as, to alternate representations of the same resource. This Standard does not mandate any particular approach how this is supported by the server.

This merely states that if a server provides a type-specific link (e.g., for rel="self" and "alternate"), then there had better be something in the URL that tells the server to return that particular format regardless of the value of the HTTP Accept header. It doesn't promote type-specific links in any way.

So it seems to me that in general, links (with the exception of "self" and "alternate", which are intrinsically typed) should be typeless so that content negotiation can be performed with the HTTP Accept header. The contrasting approach would be to provide a type-specific link for each content type available for each resource. Note that most if not all resources will have multiple representations (even if it's just JSON versus HTML), so that would significantly increase the number of links communicated. It would also demote the usefulness of HTTP Accept negotiation. A third option is to have a mixed approach, where some links are typeless and some are typed. But is up to the individual specifications to dictate which should be which, or should all clients be equipped to handle both methods of content negotiation?

Our being-developed client code does the safe thing by following a type-specific link if it's of a preferred type, or following a typeless link otherwise, sending an appropriate HTTP Accept header either way. But is this the required client behaviour? If it's not, then the correct way of performing content negotiation when following links needs to be nailed down more.

To throw a wrench into the works, requirement 15 of "OGC API - Features - Part 1: Core", regarding the rel="items" links of a collection, states:

All links SHALL include the rel and type properties.

So these particular links are not allowed to be typeless. Therefore, our sample output above is technically incorrect, and should have 19 type-specific "items" links rather than 1 typeless one. But do the other links have the same requirement? E.g., Should we report separate HTML, GML and JSON links for rel="describedBy", and 7 different image formats for rel=map?

[jerstlouis]

@pomakis

with the exception of "self" and "alternate", which intrinsically demand specific content types

I also wonder whether it would be proper for a "self" link to include a typeless version, e.g. if a large number of alternative formats are supported?

A third option is to have a mixed approach, where some links are typeless and some are typed. But is up to the individual specifications to dictate which should be which, or should all clients be equipped to handle both methods of content negotiation?

My opinion is that all OGC API specifications should follow the same common pattern, and since both approaches have been used extensively, by now this would mean clients should be ready for both the link to a specific type, as well as typeless links supporting negotiation.

I agree that in general typeless links seem like a better approach, but the typed links provide the advantages of advertising the available formats provided by the server. How to do this seems less nailed down for the content negotiation approach. Clients should just just provide the list of all formats they support, but that might be quite extensive. I think there was also discussions on how HEAD and/or OPTIONS could help with this.

All links SHALL include the rel and type properties.

I wonder if this could be relaxed in a future version of Features? I checked and don't see any similar mention in Common - Part 1 now.

But do the other links have the same requirement? E.g., Should we report separate HTML, GML and JSON links

I believe typeless links should be supported and encouraged so as to avoid this proliferation of links.

rel="describedBy",

Note that describedby is all lowercase, both our own server and the Common specifications previously made the same mistake so I try to point this out everytime I see this!

[rob-metalinkage]

Without prejudicing a simplified solution that covers 80% of cases, there is a more general solution that is completely flexible and supports full discoverability of capabilities - developed to support the more demanding requiremenrts of the OGC Definitions Server as it moves towards serving full data models (application schemas) and interoperability with a wider world requiring different data profiles... https://www.w3.org/TR/dx-prof-conneg/

OGC-APIs supporting prof-conneg have been implemented in testbeds.

If possible, (and if not why not), the OGC API should allow (i.e. not mandate incompatible mechanisms) for prof-conneg to be implemented on top of OGC API to support the more complex cases you may find and NOT re-define equivalent functionality.

For the end user the proliferation of different ways to do the same thing creates a dramatic INCREASE in complexity, at odds with the typical intent to simplify by adding a special purpose mechanism. We have an opportunity to have the simplest possible solution for the majority of cases and a single powerful extension mechanism. (and if there is an issue with the power of dx-prof-conneg, or an idea to make it easier to use, please raise that as an issue - it can be explored)

[ghobona]

@rob-metalinkage Content Negotiation by Profile appears to be a draft specification and has not yet been approved as a W3C Recommendation. Any idea when it is likely to be completed/approved as a W3C Recommendation?

[jerstlouis]

@rob-metalinkage Content negotiation by profile is the focus of #8 .

I may not fully understand your suggestion here, but it seems to me like this is an orthogonal issue.

[cmheazel]

We have to support everything from the thick desktop client to follow-the-link hypermedia. There is no good solution which fits all. So we have provided implementors the flexibility (within bounds) to do what is appropriate for their implementation. Typed links with content negotiation through the accept header is preferred. But that is not always possible (think hyperlinks in an OGC standard). So we also allow the -f query parameter and minted URIs.

This section of API Common Part 1 may not be a clear as it should be. I'll see if I can clean it up.

[jerstlouis]

@cmheazel

Typed links with content negotiation through the accept header is preferred.

Wouldn't this be contradictory?
If a link is to a specific type, then content negotiation does not work (the f query parameter overrides the accept header).

[cmheazel]

@jerstlouis You are perfectly free to set the link type to XML, the -f parameter to JSON, and the suffix to .HTML. And you are free to explain to your users why you did such a foolish thing.

[pomakis]

You are perfectly free to set the link type to XML, the -f parameter to JSON, and the suffix to .HTML. And you are free to explain to your users why you did such a foolish thing.

... as long as the response is XML. There are really only two modes: a typed-link mode where the response content type is specified by the link (regardless of the HTTP Accept header, and often driven by a vendor-specific "f" parameter), and a typeless-link mode where the response content type is negotiated by the HTTP Accept header.

[m-mohr]

As a client implementor I want to retrieve all supported file formats upfront so that I can let users choose them for e.g. Coverage Download or Processing purposes, which seems to be a OGC API wide weakness according to a recent discussion with @jerstlouis. I'd like to encourage to provide a solution for this in OGC APIs, otherwise it will be very diffiult to implement user-freiendly clients.

All the details can be found in the following discussion: opengeospatial/ogcapi-coverages#166 (comment)

[cportele]

@m-mohr

The user-friendly client will include the formats that it can handle in the "Accept" header, with weights to express preferences, and the server will return the best format it can serve - or a 406 response, if it cannot support any of the requested formats. Some servers may also send just any format back, if there is no match, so the client should check the "Content-Type" to see, if it can handle the returned format. This is what browsers do, too.

If you consider this an architectural weakness, then you may want to raise this with the IETF HTTP WG. This behavior is part of the architecture of the web and the OGC API standards are based on that architecture.

The second approach for content negotiation supported by HTTP is that the server returns a 300 response with all the different representations that the server supports (e.g., if there are no headers for server-driven content negotiation). However, there are no rules how the response lists the representations so that does not make it easier for the client and - probably as a result - I don't think 300 responses are used much. Browser support is also a mixed bag not to mention other clients.

Alternatively, you could send a HEAD request to the resource and check the "Link" header(s). For OGC API resources the standard approach is to return "self" and "alternative" links with the "type" attribute. For example:

% curl -I "https://demo.ldproxy.net/vineyards/collections/vineyards/items/410345"
HTTP/2 200 
content-bounding-box: 7.022685, 49.914455, 7.040222, 49.920208
content-crs: <http://www.opengis.net/def/crs/OGC/1.3/CRS84>
content-disposition: inline; filename="410345.csv"
content-language: en
content-type: text/csv
date: Wed, 06 Sep 2023 18:07:03 GMT
etag: "52d4f8eab4ee7c41cbb6cd275599d5ba"
link: <https://demo.ldproxy.net/vineyards/collections/vineyards/items/410345?f=fgb>; rel="alternate"; title="This document as FlatGeobuf"; type="application/flatgeobuf"
link: <https://demo.ldproxy.net/vineyards/collections/vineyards/items/410345?f=html>; rel="alternate"; title="This document as HTML"; type="text/html"
link: <https://demo.ldproxy.net/vineyards/collections/vineyards/items/410345?f=json>; rel="alternate"; title="This document as GeoJSON"; type="application/geo+json"
link: <https://demo.ldproxy.net/vineyards/collections/vineyards/items/410345?f=jsonfg>; rel="alternate"; title="This document as JSON-FG"; type="application/vnd.ogc.fg+json"
link: <https://demo.ldproxy.net/vineyards/collections/vineyards/items/410345?f=jsonfgc>; rel="alternate"; title="This document as JSON-FG (GeoJSON Compatibility Mode)"; type="application/vnd.ogc.fg+json;compatibility=geojson"
link: <https://demo.ldproxy.net/vineyards/collections/vineyards?f=json>; rel="collection"; title="The collection the feature belongs to"; type="application/json"
link: <https://demo.ldproxy.net/vineyards/collections/vineyards/items/410345?f=csv>; rel="self"; title="This document"; type="text/csv"
vary: Accept,Accept-Language,Accept-Encoding
x-request-id: 7e1697fe-8816-4903-8e4b-4f7e1e28fd4e
content-length: 265

[m-mohr]

Thanks.

I'm just looking for a very simple use case in a Web UI, for example in Coverages or Processes. I want to let the user choose the file format they wish to retrieve for download. As such the client supports all formats, i.e. Accept: */*. I assume letting the user type in a media type into a free-form test box is not a good UI.

How can I make this possible? I don't see a reliable solution yet. It sounds like the HEAD request (with Accept: */*?) would be a solution, but it seems pretty equivalent to just reading the links with rel type coverage from the Collections for example.
There you have the issue that it looks like the type field is not required though, so sometimes you encounter multiple links with different types (good) or one line with no type (bad).

I'm facing this in practice right now so. I'm implementing such a client and struggle a lot with the approach OGC has taken here. Maybe there's a reasonable solution to it, but I haven't found it yet.

[jerstlouis]

@cportele The HEAD request is not currently a requirement of the OGC API standards yet though, right? It is only a recommendation in Features (recommendation 3 in the latest Editor's Draft).

[cportele]

@m-mohr

As pointed out, it os not an OGC approach, it is the approach of the Web / IETF. If that turns out to be an issue for a wider range of clients in an OGC context (but I do not think this is clear yet), then we could think about an additional requirements class that supports such a capability.

There were previous discussions to use OPTIONS to discover capabilities of a resource, but since others do not seem to face that issue, OPTIONS is typically only used for the supported methods, so this was not pursued further so far.

@jerstlouis

Correct, HEAD support and also including "Link" headers are only recommended, not required.

[m-mohr]

If OGC APIs are primarily targeting M2M use cases then it's fine as it is. If you want user-friendly clients for humans* that adopt to the underlying API then you'll need more than what you have right now. The GDC Editor currently developed in the Testbed shows this nicely. Somethine like HEAD support with Links would be required for that, but I did't see that being implemented anywhere.

* An average user that is not so technically oriented and as such is not into media types