Skip to content

Support incompatible_compact_repo_mapping_manifest in runfiles library #3022

Open
@rickeylev

Description

@rickeylev

Under bzlmod, the repo mapping can become quite large (i.e. tens of megabytes) because its size scales as a factor of the number of repos in the transitive dependencies.

To address this, the --incompatible_compact_repo_mapping_manifest flag was introduced. This changes the repo mapping formation to use prefixes (instead of exact repo names) for mapping things.

To make this work with the runfiles library, the code has to be updated to handle these prefixes instead of just exact strings.

# old:
+deps+dep1,aaa,xxx
+deps+dep2,aaa,xxx

# new
+deps+*,aaa,xxx

Order matters (i.e. an earlier prefix should have precedence).

For reference, the names of these elements are source_canonical,target_apparent,target_canonical (i.e. "Within the canonical name @source, lookups for @target_apparent result in @target_canonical")

This looks relatively straight forward and the Java impl should translate over nicely. Dicts are insertion-ordered in Python as of Python 3.7 (long past EOL).

Core algo that needs to go into runfiles.py


def parse_repo_map(entries):
  exact = {}
  prefixed = {}
  for s, ta, tc in entries:
    if s.endswith("*"): prefixed[(s, ta)] = tc
    else: exact[(s, ta)] = tc
  return exact, prefixed

def rlocation(source, target):
   if (source, target) in exact: return exact[(source, target])
   for (prefix, ta), tc in prefixed.items():
     if ta == target and source.startswith(prefix): return tc

Refs:

cc @fmeum

Metadata

Metadata

Assignees

No one assigned

    Labels

    Good first issueA good first issue for people looking to contributecleanupTech debt, resolving it improves our own velocitycore-rulesIssues concerning core bin/test/lib rulestype: feature request

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions