Add support for JSON-LD structured data

[hparra]

This discussion proposes adding support for JSON-LD-based structured data. I support SEO Tech for Adobe.com and am available to make the proposed changes if all stakeholders agree.

Goals:

1. Authors can add multiple instances of JSON-LD of any schema to their documents using the metadata block (or a new metadata-like block)
2. Webmasters can use this JSON-LD data in their search indices using query-index.yaml

Non-goals:

• Ideal Authoring UX for structured data
• Multiple JSON-LD metadata blocks specific to schema type

Features

Authors can add any type of JSON-LD data in a rudimentary way using the metadata block:

If we (developers) prefer, we can make a new metadata-like block instead:

Webmasters can index on JSON-LD data. Example:

indices:
  videos:
    properties: 
      title:
        select: head > script[type="application/ld+json"].videoObject
        value: |
          jsonQuery(textContent(el), 'title')
      description:
        select: head > script[type="application/ld+json"].videoObject
        value: |
          jsonQuery(textContent(el), 'description')

This search index can also act as an SEO report to understand which pages have videos and corresponding structured data.

Proposed Changes

• add support for JSON-LD metadata or block (html pipeline)
• add query function to handle json parsing/querying (indexer)

Alternatives Considered

Milo has developed custom key/value blocks for complex structured data (see event, video, etc.) In these cases the JSON-LD is added at render time. Google supports dynamically-generated JSON-LD but Franklin's indexer does not. This results in bad webmaster/developer UX because indexer does not support XPath either and trying to parse this data out of div tables requires regex that is unpleasant to write and difficult to get right, and would be even worse if tables just accepted a JSON string like the one proposed herein.

Blocks or autoblocks that fetch API data to create JSON-LD were proposed, but these require a call at load time (performance) to a pre-existing API (availability) that may require a credential (security). This may work for something like video, but such a feature may be fragmented depending on its source (e.g. YouTube vs. AdobeTV vs. B.Y.O. Video). We could create a simple API using Franklin's spreadsheet features, but it does not support search, and we may be loading a huge JSON just to do so (performance). The indexer problem would still exist.

All proposals, including this one, have less than ideal authoring UX. They all include significant and/or cryptic data entry, but this proposal enables the most. An improved UX is beyond the scope of this proposal, but in the future I would like to propose features in Sidekick to help with the creation of pre-filled blocks and/or database-driven batch jobs that add the metadata/block to the document.

Conclusion

JSON-LD is used to represent document metadata, just like meta tags do. Authors should be able add JSON-LD using any type of schema without restriction. Webmasters should be able to use JSON-LD for their search indices just like Google does with theirs. This proposal enables both.

[tripodsan]

I don't think that this is very author friendly and looks rather complicated for authors to write JSON-LD blocks correctly.
why not just split the individual properties into a table and let the author fill them in in a natural way?

[hparra]

I don't think that this is very author friendly and looks rather complicated for authors to write JSON-LD blocks correctly.

It's not user-friendly, but in this case it's better to enable than to restrict. JSON-LD is being given more relevance weight and search results real-estate by Google. Authors want their content to be found.

Authors will find a way to create JSON-LD using various tools, examples: 1, 2, 3.

Sidekick can be used to improve the Authoring experience here, and I'm ready to work with PM to make that proposal.

why not just split the individual properties into a table and let the author fill them in in a natural way?

JSON-LD is not flat. These are complex objects with child objects, arrays with objects, etc. See examples of your suggestion mentioned in "Alternatives Considered" (event, video). There is still authoring complexity there.

[trieloff]

Hard no on "less than ideal authoring UI". If we can't nail the authoring, there is no chance this feature will be used by authors.

[hparra]

If we can't nail the authoring, there is no chance this feature will be used by authors.

Authors cannot use a feature that does not exist. We are restricting, not enabling, by forcing authors to have a developer that will make blocks or autoblocks that will do this for them.

Sidekick can be used to improve the Authoring experience here, and I'm ready to work with PM to make that proposal, including how to measure success. If numbers show that no one really uses, then we can remove feature.

[kptdobe]

But nobody wants to author a LD schema! Capturing the data should be done in a use friendly matter and if really needed, the schema can be generated client side (we do that for some blocks in Express https://github.com/adobe/express-website/blob/main/express/blocks/how-to-steps-carousel/how-to-steps-carousel.js#L209-L212).
And from an index perspective... there is no benefit to index properties via the schema object while you can easily access the value via the page metadata.

[hparra]

An Author who wants their content found will do what is necessary. Authors will still find a way to create JSON-LD using various tools, examples: 1, 2, 3. We should not prevent them from doing so.

Sidekick can be used to improve the Authoring experience here, and I'm ready to work with PM to make that proposal.

Capturing the data should be done in a use friendly matter and if really needed, the schema can be generated client side (we do that for some blocks in Express https://github.com/adobe/express-website/blob/main/express/blocks/how-to-steps-carousel/how-to-steps-carousel.js#L209-L212).

We are doing that as well. See mentioned PRs mentioned in "Alternatives Considered": event, video.

And from an index perspective... there is no benefit to index properties via the schema object while you can easily access the value via the page metadata.

We can't access some values easily from page metadata approach. JSON-LD is not flat. These are complex objects with child objects, arrays with objects, etc. We can't flatten these into keys that are any more intuitive, or make the data entry much better. See PR for previous version of event for example of drawbacks to page metadata approach and why it was rejected.

[rofe]

is there any downside to simply adding JSON-lD to the DOM client side? As the example shared by @kptdobe shows we've done this successfully for Express...

[hparra]

is there any downside to simply adding JSON-lD to the DOM client side?

We can't select data for index property easily unless we add it to the markup. We cannot make assumption that we are adding same data in JSON-LD to other parts of the original document. See first paragraph of "Alternatives Considered" for more.

As the example shared by @kptdobe shows we've done this successfully for Express...

We are doing that as well. See PRs mentioned in "Alternatives Considered": event, video. We're unsure this authoring experience is much better and we lose the simple index properties.

[hparra]

Spoke to @davidnuescheler today and he concurs with majority opinion.

[herzog31]

Adding those properties client-side might not be an option, at least for product data. For regular Google search this seems fine.

Product data:

Added structured data markups can appear anywhere in the HTML source. Any added markups must be present in the HTML returned from the web server, and can’t be generated after a page has been loaded using JavaScript.

Source: https://support.google.com/merchants/answer/7331077?hl=en#zippy=:~:text=Added%20structured%20data%20markups%20can%20appear%20anywhere%20in%20the%20HTML%20source.%20Any%20added%20markups%20must%20be%20present%20in%20the%20HTML%20returned%20from%20the%20web%20server%2C%20and%20can%E2%80%99t%20be%20generated%20after%20a%20page%20has%20been%20loaded%20using%20JavaScript.

Regular search:

A JavaScript notation embedded in a <script> tag in the  and  elements of an HTML page. The markup is not interleaved with the user-visible text, which makes nested data items easier to express, such as the Country of a PostalAddress of a MusicVenue of an Event. Also, Google can read JSON-LD data when it is dynamically injected into the page's contents, such as by JavaScript code or embedded widgets in your content management system.

Source: https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data#:~:text=A%20JavaScript%20notation,content%20management%20system

[hparra]

This was added in adobe/helix-html-pipeline#653