RFE: expose delegated metadata to client application

[jku]

EDIT: The overall issue is described in detail in https://docs.google.com/document/d/1rWHAM2qCUtnjWD4lOrGWE2EIDLoA7eSy4-jB66Wgh0o . The suggestion here is roughly the Metadata role (file) as search index solution in the document.

Assume a setup like this (this is what we expect a community artifact repository like PyPI to look like if it uses developer signatures with TUF):

• a specific project/product team controls a delegated metadata
• TUF clients want to know details of all of the artifacts in this metadata (to e.g. figure out which versions of an artifact are available)

Currently there is no way for the client application to get the whole metadata content from ngclient. We could provide a call much like get_targetinfo() that instead of the TargetFile would return the Targets object where the target search ended:

def get_targets_metadata(target_path: str) -> Targets
    """returns a Targets object of the metadata where the search for target_path terminated"""

This is not applicable to every TUF repo:

• it requires a "contract" between repository and client: client has to know of a target_path that is delegated to the correct metadata -- in the pypi example it could be e.g. the PyPI project name
• this is only useful if all "related" target files are listed in the same metadata

But with those assumptions the client can now easily get not just the list of target files it's interested in but also any custom metadata embedded in the targets metadata.

I've not thought through all the cases (what happens if there is no targetpath match? what if there is no terminating delegation?) but I think this is something we could consider implementing

[JustinCappos]

I would expect that in the case that this target file delegates to other target files, it would include all of the items listed there. (This would continue transitively.) It would also need to handle special delegation cases, thresholds, etc. Is this your thinking as well?

[jku]

I was not thinking that, no -- but that could work as well...

My original idea was to literally return the equivalent of the "signed" json object of the "final" targets metadata (the one that terminates delegation either by containing the targetpath or by terminating=True): this would require no extra processing on the client part, but as I mentioned is only useful if all "related" target files are listed in the same metadata. Your idea of building a list of target files while doing the depth first search through the delegation tree (and appending all target files iff the targets metadata happens to be part of the delegated portion of the tree) is certainly more complex but it is interesting as it removes the limitation of "one metadata for related target files" -- I would have to prototype to see if there are any unintended results there.

This is what I proposed:

def get_targets_metadata(target_path: str) -> Targets
    """returns a Targets object of the metadata where the search for target_path terminated"""

This what I think Justin is describing:

def get_all_targetinfos(target_path: str) -> List[TargetFile]:
    """Returns a list of all target files in all targets metadata that forms the delegating chain for 'target_path'"""

I don't think there's anything special to handle wrt threshold etc: if the delegations work for normal targetpath search, they should work for this.

[dennisvang]

@jku If you are looking for another use case: it looks like our notsotuf client would also benefit from such a feature.

[jku]

@dennisvang it may have been your comments some time ago that got me thinking about it :)

Btw if you have any feedback or suggestions on python-tuf 1.x from downstream perspective, that would be very welcome -- creating issue is fine or slack works too

[lukpueh]

This is what I proposed:

def get_targets_metadata(target_path: str) -> Targets
    """returns a Targets object of the metadata where the search for target_path terminated"""

This what I think Justin is describing:

def get_all_targetinfos(target_path: str) -> List[TargetFile]:
    """Returns a list of all target files in all targets metadata that forms the delegating chain for 'target_path'"""

I think this is a useful feature! Here are some unordered thoughts/questions on the two existing proposals:

• Both solutions require a contract between repo and client that the function yields all/only "related" target file infos.
• Solution 2 seems more general/flexible as it can cover cases, where related target file infos are spread across multiple target metadata files AND where they are all in the last target file. This seems like an advantage.
• Is solution 2 more prone to also serve unrelated target file infos? Probably not / depends on the "contract".
• Why does solution 1 return a full Targets object and solution 2 a list of only TargetFiles?
• Do we need the full Targets metadata in either case?
• Is target_path here the same as in get_targetinfo or can it also be a path prefix or path pattern?
• If target_path can be only a part of the path or a pattern, then the delegation tree might not resolve in the same way as the individual "related" files would.

[dennisvang]

@jku issue #822 looks related.

[jku]

Yes it definitely is related. Searching is still a very complex beast and we shouldn't think this will actually solve that problem completely: I don't think this library even can solve searching in general: it really is an application problem. But we could provide this functionality so that repositories can design their content so that this functionality can be used for specific types of searches.

[jku]

Forgot to respond to lukas here:

Is solution 2 more prone to also serve unrelated target file infos? Probably not / depends on the "contract".

Yeah, there's certainly a chance of an earlier targets metadata to contain "unrelated" files that get listed (in the same sense that it allows multiple metadata to contain the "related" files). This is what makes the two approaches different...

Why does solution 1 return a full Targets object and solution 2 a list of only TargetFiles?
Do we need the full Targets metadata in either case?

solution 1 returns Targets just because it can -- I figured this would allow e.g. custom fields in the Targets to be available to client. In the second option it's not as simple.

I don't know of a specific need for Targets.

Is target_path here the same as in get_targetinfo or can it also be a path prefix or path pattern?

I think it has to be the same thing: an explicit targetpath that in this case is just used to find "all targetfiles in the chain of delegations for this targetpath" (or "...in the last delegation for this targetpath" for solution 1). It's a bit unintuitive but could be useful...

[jku]

Do we need the full Targets metadata in either case?

Oh and the opinion I forgot: I maybe lean towards the List[TargetFile] return value anyway regardless of solution. ngclient public API already includes TargetFile, but does not currently expose the Signed-derivatives or other Metadata API details: I like that split

[dennisvang]

The List[TargetFile] option would be sufficient for our specific use-case.

Currently, we base our search on the target_path values obtained from tuf.ngclient.updater.Updater._trusted_set.targets.signed.targets.keys(), although I'm not sure that would always work.