Mirroring

[Geobert]

Hi again!

Opening a new discussion for a new topic :)

How the mirroring feature is supposed to work?

The Meuse doc is saying we need to fork the crates.io index, that would work at the time of forking, but then, how to discover new crates?

I tried to setup a mirror index with this config.json:

{"dl":"https://172.18.0.2/ktra/api/v1/mirror","api":"https://172.18.0.2"}

IP here is the one of the docker container

Then followed the Meuse's doc:

[source.crates-io]
replace-with = 'mirror'

[source.mirror]
registry = "https://github.com/mcorbin/crates.io-index"

Obviously, registry is another value for me as it is on our private git instance

I started with an empty mirror index but get this error:

$ cargo check
    Updating `https://github.com/mcorbin/crates.io-index` index
error: no matching package named `libc` found
location searched: registry `https://github.com/rust-lang/crates.io-index`

again, the url has been replaced here for privacy reason.

What am I doing wrong?

[moriturus]

Sorry for lack of documentation about the mirroring.
I have to write the book about the feature but I am too busy to do so.

The steps to use the mirroring feature are:

1. Fork the crates.io index repository.
2. Edit or create your cargo config similarly to the Meuse's way.

In your case, the step 2 is fine but it is wrong to start with an empty index repository.
You have to start with forked one.

Additionally, you have to rebase your forked crates.io index to the original latest one by yourself to discover new crates.

[Geobert]

No worries about the doc, I was expecting a similar way to Meuse as it was the link I send to you :)

Doesn't this way of doing a mirror a bit cumbersome?

What about this idea:

We configure cargo to only hit ktra.

• if a crate exists, then serve it
• if not, fetch it on crates.io, cache it and serve it

What do you think?

[moriturus]

I think that to manage a forked index repository is not ktra's business to avoid complications about name-space conflicts. Meuse takes the same approach as far as I guess.

But this is only for the index repository, crate files are cached if you hit the mirroring API even once.

[Geobert]

OK I can understand why you want to avoid to go this route :) name conflicts can be a pain to manage indeed.

What's bothering me from a user perspective is to rebase the mirror index manually.
Would you consider some automation of the rebasing?

If not, I'll look into a cron script to do that externally from ktra :)