Allow for additional Metadata.xlsx files

[Blainegunn]

Over on the DC site we'd like the option to have multiple Metadata.xlsx files. The situation is the Metadata.xlsx file currently is used by engineering and web publishing. There have been some cases where content is prematurely publishing since our teams aren't on the same schedule. Ideally we'd like 2 Metadata.xlsx files so engineering can publish the items they need without also making the web publishing content live.

[davidnuescheler]

i think that's a good idea. @rofe @tripodsan, i think we have seen issues with the centralized metadata and multiple people editing / publishing things. i wonder what's the right way of breaking this up... also i think that the structure of the metadata sheet is probably subject to be reviewed anyway, as it usually is very sparse.

[dylandepass]

We could potentially follow the same pattern as config where multiple configs (config.json and headers.json) are merged into config-all.json. The naming of these multiple metadata files I’m not sure.. We could just merge any json file with metadata- in the name into metadata-all.json.

@davidnuescheler We did address the "sparse" issue with https://github.com/adobe/project-helix/discussions/642. This is live today for metadata.

[tripodsan]

I would not generate a metadata-all.json but just add all the data to the config-all.json. I don't think there is a need (or even a desire) to make the metadata.json available on the client.

We could just merge any json file with metadata- in the name ...

+1

[Blainegunn]

@davidnuescheler thank you.
Just for the sake of planning any thoughts on when this could potentially be ready?
thanks again!

[Blainegunn]

Proposal to Support Multiple Metadata Spreadsheets

Justification:

Some of the ACOM migration projects require Metadata.xlsx be used by web producers, QAs and developers. All for different reasons. The issue this presents is staged content is being published prematurely. The thought is if we could have additional sheets we could avoid these mishaps.

Naming Conventions:

Metadata.xlsx is the standard. Additional metadata sheets must be hyphenated like this:
Metadata-promos.xlsx
Metadata-dcweb.xlsx

Duplication:

Metadata.xlsx will continue to be the main spreadsheet and source of truth. If there’s duplication in any other metadata sheets the data in Metadata.xlsx should override it.
If there is duplication in any of the additional sheets the sheet last previewed/published should be honored.

[Blainegunn]

@rofe

[tripodsan]

come to think of it, i'd rather have a distinct list in the /.helix/config.json of all the metadata sheets that are applied in the specified order. especially if some technical metadata never should be overwritten by an authored property, you probably want to apply the technical metadata last.

also, I'm afraid that a generic metadata-* approach might need to abuse (e.g. have a metadata for each document, ala page properties). so having a distinct list would prevent that.

also, this would prevent potential problems with stray metadata sheets somewhere in the drafts....

another approach would be, that there can be only 1 metadata.xlsx per folder, and automatically limit the scope of the metadata to below that folder. but this would probably not do the initial requirement justice.

[rofe]

I like the approach where we have an explicit, ordered list in the config sheet best, I think it would be most intuitive and transparent.

We'd still have to define where folder-mapped metadata sheets would fit in. Today, (I think) the order is:

1. Page level metadata block wins over
2. Folder-mapped metadata sheet wins over
3. Global metadata sheet

If we now split the global metadata into multiple sheets, would it become:

1. Page level metadata block wins over
2. Folder-mapped metadata sheet wins over
3. Metadata sheet(s) in configured order wins over
4. Global metadata sheet

Or would the global metadata sheet become part of the configured metadata sheets?

[tripodsan]

yes. the later.

the folder mapped metadata is read during delivery, the extra sheets would be applied to the the overall global metadata during preview.

[3ch023]

Hi @tripodsan @rofe , is there a limit on how many additional metadata files the project can have?

Context

on adobe.com we have a use case when the metadata needs to be added to the page without touching the original document.
One option to do this would be using global metadata, but for Content Authors it's too risky to have one file affecting multiple pages in all locales.
Because of that, they are requesting one metadata file per locale, which would result in 90+ additional metadata files.

• Is there any risk/performance impact with having this many additional metadata files?

• Is there maybe a better way to attach some metadata to the page without the need to publish the original document?

[tripodsan]

• Is there any risk/performance impact with having this many additional metadata files?

well, there are 2 sides to this:

1. the system updates the internal global metadata file from all the individual ones, when one of them is modified. this should not be a performance problem.
2. the other issue might be, that if you eventually have a large amount of metadata, for example, having 10k entries for 90+ locales could result in a very large internal global metadata file, which is loaded on every .page request and then applied to the currently requested page.

I think for now, this should not be a problem, but we might need to keep an eye on the sizes.

• Is there maybe a better way to attach some metadata to the page without the need to publish the original document?

currently not.

[3ch023]

thank you @tripodsan

for now, I don't anticipate that many pages to be involved, we unlikely will cross 2k. But I will keep this limitation in mind. What I just realized, it doesn't even make the difference if we split the file or not - in any case, the large number of entries will affect the rendering

[tripodsan]

in any case, the large number of entries will affect the rendering

we need to test, but probably not much slower.