Skip to content

Commit

Permalink
Document type-piracy / type-leakage restrictions for require_stdlib (
Browse files Browse the repository at this point in the history
…#56005)

I was a recent offender in
JuliaLang/Pkg.jl#4017 (comment)

This PR tries to lay down some guidelines for the behavior that stdlibs
and the callers of `require_stdlib` must adhere to to avoid "duplicate
stdlib" bugs

These bugs are particularly nasty because they are experienced
semi-rarely and under pretty specific circumstances (they only occur
when `require_stdlib` loads another copy of a stdlib, often in a
particular order and/or with a particular state of your pre-compile /
loading cache) so they may make it a long way through a pre-release
cycle without an actionable bug report.
  • Loading branch information
topolarity authored Oct 6, 2024
1 parent 43f4afe commit 2ca88ad
Showing 1 changed file with 40 additions and 0 deletions.
40 changes: 40 additions & 0 deletions base/loading.jl
Original file line number Diff line number Diff line change
Expand Up @@ -2610,6 +2610,46 @@ function _require_from_serialized(uuidkey::PkgId, path::String, ocachepath::Unio
end

# load a serialized file directly from append_bundled_depot_path for uuidkey without stalechecks
"""
require_stdlib(package_uuidkey::PkgId, ext::Union{Nothing, String}=nothing)
!!! warning "May load duplicate copies of stdlib packages."
This requires that all stdlib packages loaded are compatible with having concurrent
copies of themselves loaded into memory. It also places additional restrictions on
the kinds of type-piracy that are allowed in stdlibs, since type-piracy can cause the
dispatch table to become visibly "torn" across multiple different packages.
The specific requirements are:
The import side (caller of `require_stdlib`) must not leak any stdlib types, esp.
to any context that may have a conflicting copy of the stdlib(s) (or vice-versa).
- e.g., if an output is forwarded to user code, it must contain only Base types.
- e.g., if an output contains types from the stdlib, it must be consumed "internally"
before reaching user code.
The imported code (loaded stdlibs) must be very careful about type piracy:
- It must not access any global state that may differ between stdlib copies in
type-pirated methods.
- It must not return any stdlib types from any type-pirated public methods (since
a loaded duplicate would overwrite the Base method again, returning different
types that don't correspond to the user-accessible copy of the stdlib).
- It must not pass / discriminate stdlib types in type-pirated methods, except
indirectly via methods defined in Base and implemented (w/o type-piracy) in
all copies of the stdlib over their respective types.
The idea behind the above restrictions is that any type-pirated methods in the stdlib
must return a result that is simultaneously correct for all of the stdlib's loaded
copies, including accounting for global state differences and split type identities.
Furthermore, any imported code must not leak any stdlib types to globals and containers
(e.g. Vectors and mutable structs) in upstream Modules, since this will also lead to
type-confusion when the type is later pulled out in user / stdlib code.
For examples of issues like the above, see:
[1] https://github.com/JuliaLang/Pkg.jl/issues/4017#issuecomment-2377589989
[2] https://github.com/JuliaLang/StyledStrings.jl/issues/91#issuecomment-2379602914
"""
function require_stdlib(package_uuidkey::PkgId, ext::Union{Nothing, String}=nothing)
@lock require_lock begin
# the PkgId of the ext, or package if not an ext
Expand Down

0 comments on commit 2ca88ad

Please sign in to comment.