Combining regular and weak dependencies for testing purposes

[maleadt]

I want to test NNlib.jl, which has a weak dep on CUDA.jl and cuDNN.jl, using my development version of CUDA.jl/cuDNN.jl. This does not seem to work, presumably because of the weakdep.

$ git clone [email protected]:FluxML/NNlib.jl.git NNlib

$ cat NNlib/Project.toml
[deps]
# no CUDA or cuDNN
[weakdeps]
CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba"
cuDNN = "02a925ec-e4fe-4b08-9a7e-0d78e3d38ccd"

$ jl --project=NNlib
(NNlib) pkg> dev CUDA cuDNN
Path `/home/tim/Julia/pkg/CUDA` exists and looks like the correct repo. Using existing path.
    Updating `~/Julia/pkg/NNlib/Project.toml`
  [052768ef] + CUDA v5.5.1 `~/Julia/pkg/CUDA`
  [02a925ec] + cuDNN v1.4.0 `~/Julia/pkg/CUDA/lib/cudnn`
    Updating `~/Julia/pkg/NNlib/Manifest.toml`
  [052768ef] + CUDA v5.5.1 `~/Julia/pkg/CUDA`
  [02a925ec] + cuDNN v1.4.0 `~/Julia/pkg/CUDA/lib/cudnn`

$ cat NNlib/Project.toml
[deps]
CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba"
cuDNN = "02a925ec-e4fe-4b08-9a7e-0d78e3d38ccd"
[weakdeps]
CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba"
cuDNN = "02a925ec-e4fe-4b08-9a7e-0d78e3d38ccd"

So far so good. The ]st output looks confusing though, with no trace of CUDA or cuDNN:

$ jl --project=NNlib
(NNlib) pkg> st
Project NNlib v0.9.24
Status `~/Julia/pkg/NNlib/Project.toml`
  [79e6a3ab] Adapt v4.0.4
  [a9b6321e] Atomix v0.1.0
  [d360d2e6] ChainRulesCore v1.25.0
  [46192b85] GPUArraysCore v0.1.6
  [63c18a36] KernelAbstractions v0.9.27
  [37e2e46d] LinearAlgebra
  [9a3f8284] Random
  [10745b16] Statistics v1.10.0

I guess that may be because it's both a dep and a weakdep, which is weird. The bigger problem however is that testing discards my development versions of CUDA.jl and cuDNN.jl:

$ jl --project=NNlib
(NNlib) pkg> test
     Testing NNlib
      Status `/tmp/jl_p6O4zu/Project.toml`
# no mention of CUDA or cuDNN

Adding CUDA or cuDNN to the test Project.toml doesn't help either; it just picks up the released versions.

I realize that this is a peculiar combination of dependencies, but it does seem impossible right now to test an unreleased version of CUDA.jl with NNlib.jl.

Tested on 1.10.

[vchuravy]

I discussed the same situation recently with @KristofferC on Slack, he mentioned that this should be fixed on 1.11, but in my testing it was only fixed in 1.12

But even though in 1.12 it works by adding CUDA to the deps and removing it from the weakdeps. This kinda breaks free for me and I would need to manually move the dependency back to weakdeps.

Crucially this works because on 1.11/1.12 deps will also trigger extensions

[maleadt]

Crucially this works because on 1.11/1.12 deps will also trigger extensions

Combining deps and extensions (instead of weakdeps) results in an error on 1.11:

❯ cat /tmp/dev/NNlib/Project.toml
[deps]
CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba"
cuDNN = "02a925ec-e4fe-4b08-9a7e-0d78e3d38ccd"

[weakdeps]
# no CUDA or cuDNN

[extensions]
NNlibCUDACUDNNExt = ["CUDA", "cuDNN"]
NNlibCUDAExt = "CUDA"

ERROR: NNlib has a malformed Project.toml, the extension package CUDA is not listed in [weakdeps]

Works on 1.12. If this is the blessed way (which still seems a bit hacky), could it be backportable to 1.11.x?

[KristofferC]

If this is the blessed way (which still seems a bit hacky), could it be backportable to 1.11.x?

Yes, we should do that.

[KristofferC]

JuliaLang/julia#54009 and #3865 should be backported.