Skip to content

Rust module : Add support for cargo subprojects #8514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

dcbaker
Copy link
Member

@dcbaker dcbaker commented Mar 10, 2021

I want to preface this that this is still a work in progress, there's a lot left to do, but I have been able to compile a couple of simple demo projects with this, so we've gotten to the point that "something works". I'm sharing this mostly to get a little visibility into the work.

The end goal is to be able to build Rust crates as fully native Meson subprojects, there's still some work to do on a lot of fronts, but it's getting there.

@dcbaker dcbaker added the module:rust Specific to the Rust module label Mar 10, 2021
@lgtm-com
Copy link

lgtm-com bot commented Mar 10, 2021

This pull request introduces 9 alerts when merging a8308e6 into 4fda4f5 - view on LGTM.com

new alerts:

  • 3 for Unused import
  • 2 for Module is imported with 'import' and 'import from'
  • 1 for Unreachable code
  • 1 for 'import *' may pollute namespace
  • 1 for Nested loops with same variable
  • 1 for Unguarded next in generator

@lgtm-com
Copy link

lgtm-com bot commented Mar 11, 2021

This pull request introduces 9 alerts when merging cc776fb into a1c8376 - view on LGTM.com

new alerts:

  • 3 for Unused import
  • 2 for Module is imported with 'import' and 'import from'
  • 1 for Unreachable code
  • 1 for 'import *' may pollute namespace
  • 1 for Nested loops with same variable
  • 1 for Unguarded next in generator

@dcbaker dcbaker force-pushed the wip/2021-01/rust-module-cargo branch from cc776fb to 0ba8ef5 Compare April 21, 2021 05:02
@lgtm-com
Copy link

lgtm-com bot commented Apr 21, 2021

This pull request introduces 11 alerts when merging 0ba8ef5 into 8fa7f3a - view on LGTM.com

new alerts:

  • 3 for Unused import
  • 3 for 'import *' may pollute namespace
  • 2 for Module is imported with 'import' and 'import from'
  • 1 for Unreachable code
  • 1 for Nested loops with same variable
  • 1 for Unguarded next in generator

@lgtm-com
Copy link

lgtm-com bot commented Apr 21, 2021

This pull request introduces 8 alerts when merging 6f14ace into 8fa7f3a - view on LGTM.com

new alerts:

  • 3 for 'import *' may pollute namespace
  • 2 for Module is imported with 'import' and 'import from'
  • 1 for Unreachable code
  • 1 for Nested loops with same variable
  • 1 for Unguarded next in generator

@jpakkane
Copy link
Member

Does someone who actually knows how this should work want to review this?

dcbaker added 21 commits May 21, 2021 10:11
I'm using a type alias for now, as I plan to change the type later in
this series
This module implements an AST building API that is somewhat nice to work
with. There's still some things here that could use cleanup, but in
general it's okay to work with.

TODO: needs some documentation
cargo is going to use ast for options, which cmake doesn't.
This is a giant blob of code. It basically reads a cargo manifest, and
then produces a meson.build from that manifest.
This adds the ability to do `import('rust').subproject('foo')`, like
cmake.
Giant blob of stuff all shoved into one giant commit
This uses pytest not unittest, unittest is such a pain in the ass and
pytest is so much nicer..
@dcbaker dcbaker force-pushed the wip/2021-01/rust-module-cargo branch from 6f14ace to 2ed446a Compare May 21, 2021 17:15
@lgtm-com
Copy link

lgtm-com bot commented May 21, 2021

This pull request introduces 8 alerts when merging 2ed446a into 619e05c - view on LGTM.com

new alerts:

  • 3 for 'import *' may pollute namespace
  • 2 for Module is imported with 'import' and 'import from'
  • 1 for Unreachable code
  • 1 for Nested loops with same variable
  • 1 for Unguarded next in generator

@lgtm-com
Copy link

lgtm-com bot commented May 21, 2021

This pull request introduces 7 alerts when merging 17f862b into 619e05c - view on LGTM.com

new alerts:

  • 3 for 'import *' may pollute namespace
  • 1 for Unreachable code
  • 1 for Module is imported with 'import' and 'import from'
  • 1 for Nested loops with same variable
  • 1 for Unguarded next in generator

@bauen1
Copy link

bauen1 commented Aug 15, 2021

Related to #2173

For fetching crates, there is cargo-download.

I was looking into setting up an embedded rust project with meson (to make it easy to integrate into C/C++ projects), and the lack of crates support is annoying. I was hacking up a way to build the crates I need with a combination of cargo download and cargo build, but this would be much better.

@dcbaker
Copy link
Member Author

dcbaker commented Aug 16, 2021

I might look at cargo download. My preference would be to not have any dependency on cargo at all, but we'll just have to see once this is actually in a state to land.

@DemiMarie
Copy link

@dcbaker I don’t think that avoiding a Cargo dependency is realistic. Cargo does a massive amount of work related to stuff like dependency resolving, some of which is even (if I understand it correctly) NP-complete (so you won’t be able to practically reimplement it in pure Python without massive performance hits).

What I do consider realistic is building Rust code that vendors its dependencies (via cargo vendor). That avoids having to interact with crates.io, which is in and of itself subtle (consider stuff like HTTP proxies, git fetching with CLI vs libgit2, etc). In short, I recommend letting Cargo do all of the network I/O; Meson can then do the actual building.

@eli-schwartz
Copy link
Member

Handling rust dependencies as subprojects means that none of this matters. Subprojects are a form of vendoring which uses pinned versions, so the only dependency resolution involved is "use it? Yes or no?" And if yes, "here is exactly one url which may be used to acquire it".

@dcbaker
Copy link
Member Author

dcbaker commented Oct 31, 2021

Right, and my plan initially is to use cargo vendor to do that vendoring. Long term i would like meson wrap to be able to handle it, which is not run at build or compile time, so the cost is paid once and then checked into git, making the cost relatively small

@Ericson2314
Copy link
Member

FYI c.f. https://github.com/kolloch/crate2nix and https://github.com/NixOS/nixpkgs/tree/master/pkgs/build-support/rust/build-rust-crate, which build Rust crates from the lockfile with little use of Cargo itself. crate2nix uses some libraries from Cargo just to be able to share data structures and parsing. buildRustCrate seems to just use cargo metadata.

@bonzini
Copy link
Collaborator

bonzini commented Dec 2, 2022

Hi @dcbaker, what is the state of this? I might be able to spend some time on it.

It seems plausible to rely solely on cargo update to build Cargo.lock + cargo vendor subprojects for the downloading, possibly mapping meson subprojects commands to cargo commands too.

@dcbaker
Copy link
Member Author

dcbaker commented Dec 2, 2022

This should really be closed in favor of #10151

The basic problem I've run into is that so many of the core types are really tied up in the interpreter, so I wrote all of the code to fix the build.py problem, and that's stalled. If you wanna help me figure out why #10714 all works fine for me but fails utterly in CI that would be super helpful :)

@dcbaker dcbaker closed this Dec 2, 2022
@bonzini
Copy link
Collaborator

bonzini commented Dec 3, 2022

Ok in the past I tried bisecting the failure, I can try doing that here too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
language:rust module:rust Specific to the Rust module
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants