beancount_parser_2 alpha

[jcornaz]

Hi,

I started a "big rewrite" of the beancount parser: beancount_parser_2

It is now in alpha. Hopefully, I'll be able to stabilize it reasonably quickly (but there's no ETA)

This old implementation (beancount-parser) is now in "maintenance mode". I'll still fix reported bugs, and I'll review/merge submitted pull requests. But I won't actively implement new features on that version.

The new implementation already has some features that the old one hasn't:

• line numbers of errors
• line numbers of directives
• more convenient to destructure the result because structs expose public fields
• can treat the account as string slices
• can use any decimal type when parsing (f64, rust_decimal::Decimal, and more)
• can catch more syntax errors (instead of silently ignoring unknown syntax)
• single result type (instead of an iterator over results)
• and more...

Though there are still many missing functionalities, you may want to have a look at it ;-)

[blais]

+Beancount ***@***.***> Looks fun. I started toying around with Rust myself. With the Python community aiming in that direction it was unavoidable I guess. Plus the C++ protobuf story just isn't getting much better. I'm wondering if there's a nicer by-ref sharing story with protos and Rust, where an instance created in Rust can be shared to Python without making copies (and possibly mutated). I'd be curious to see if your parser can be made to produce protobufs.

…

On Mon, May 22, 2023 at 9:51 AM Jonathan Cornaz ***@***.***> wrote: Hi, I started a "big rewrite" of the beancount parser: beancount_parser_2 <https://github.com/jcornaz/beancount_parser_2> It is now in alpha. Hopefully, I'll be able to stabilize it reasonably quickly (but there's no ETA) This old implementation (beancount-parser) is now in "maintenance mode". I'll still fix reported bugs, and I'll review/merge submitted pull requests. But I won't actively implement new features on that version. The new implementation already has some features that the old one hasn't: - line numbers of errors - line numbers of directives - more convenient to destructure the result because structs expose public fields - can treat the account as string slices - can use any decimal type when parsing (f64, rust_decimal::Decimal, and more) - can catch more syntax errors (instead of silently ignoring unknown syntax) - single result type (instead of an iterator over results) - and more... Though there are still many missing functionalities, you may want to have a look at it ;-) — Reply to this email directly, view it on GitHub <#38>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACSBE25P3LS3V464567Y5TXHNVN3ANCNFSM6AAAAAAYKPNV3Y> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jcornaz]

If by copying data it's not worth it. It'll be slow.

Probably indeed... Though to be honest, I am very uncomfortable talking about any performance predictions that are not backed by empirical measurements. I have been surprised too many times by my intuition being wrong. And even when right, it may not matter that much if the overhead is small enough.

Anyhow, as far as I understand, protobuf is a serialization format for exchanging data between different processes which usually do not share memory (and are often not even on the same machine). What you are talking about sounds like more like FFI to me. Is that right? Are you looking into calling some Python code from Rust within the same process?

If that's the case, then yes you should be able to interface Python without copy. But that will probably require much use of unsafe code. I am not really sure if you'll get a lot of benefits from rust in that case.

[blais]

protobuf is more than just a serialization format. The way it's used at Google it's every data representation everywhere, in files, over the network, in memory (mostly but not exclusively), in config files. You can think of an IDL with a code generator as a large slice of "the software" that gets factored out to a cross-language domain. It's a tremendously beneficial pattern of SW development, and this applies whether it's protobuf or another (ideally better) IDL + code generation system.

Yes, I'm interested in the FFI aspect, but also in the serialization aspect.

[jcornaz]

Ok, thanks for the clarification.

The way it's used at Google it's every data representation everywhere, in files, over the network, in memory (mostly but not exclusively), in config files.

I understand the use of protobuf for files, network messages and config files as they all require some kind of serialization (Though, I would personally not be enthusiastic about using protobuf for config files...).

But I've never heard of using protobuf for in-memory representation, nor for FFI. And to be honest I don't get why that would be interesting. Could you share some resources on that topic? I'd be interested to learn more about that.

[blais]

In terms of FFI, what I mean is that if you make all your data naked dumb objects (like in Beancount) what remains is just a few function calls here and there. If the vast majority of your cross-language interactiion is read/write of those objects that can account as most of your API surface. You still need some other FFI method for exposing those functions but protos will provide most of the surface calls (to access/mutate data).

[jcornaz]

Yes but, as much as I appreciate the benefit of protobuf and grpc when it comes to serialization and rpc, I don't understand what benefit protobuf brings to the table for FFI. Why not using repr(c)?

Also, I don't even understand how it is at all possible. As far as I understand, protobuf only specifies how bit are layed out in the serialized form. When not serialized, the message representation may differ between languages. And that is even (as far as I understand) one of the fundamental differences between protobuf and capnproto.

And the protobuf site says (emphasis is mine):

Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler.

So far I have been unsuccessful at finding documentation and resources on using protobuf for FFI. I'd be glad if you could share your sources.

[blais]

On Sat, Jun 10, 2023 at 4:56 PM Jonathan Cornaz ***@***.***> wrote: Yes but, as much as I appreciate the benefit of protobuf and grpc when it comes to serialization and rpc, I don't understand what benefit protobuf brings to the table for FFI. Why not using repr(c) <https://doc.rust-lang.org/nomicon/other-reprs.html#reprc>?

Because it's specific to a language (Rust).

Also, I don't even understand how it is at all possible. As far as I understand, protobuf only specifies how bit are layed out in the serialized form.

No. It also provides ser/de and bindings in many languages.

When not serialized, the message representation may differ between languages.

Yes but with Python (and maybe Rust?) you can generate bindings to an underlying C++ (or Rust?) representation. That's specifically what I'm interested in, because then I can send these data objects back and forth without incurring a copy, and this provides most of the API surface area.

And that is even, as far as I understand, is one of the fundamental difference between protobuf and capnproto <https://capnproto.org/>.

Protobuf is many things. capnproto makes a different set of tradeoffs. FWIW I'd be happy with a C++ / capnproto variant with some converters. I started poking at this some time ago.

And the protobuf site <https://protobuf.dev/> says (emphasis is mine): Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism *for serializing* structured data – think XML, but smaller, faster, and simpler. So far I have been unsuccessful at finding documentation and resources on using protobuf for FFI. I'd be glad if you could share your sources.

There's no doc. It's conceptual: if most of your API surface is shallow bindings from say, Python to an underlying C++ (or Rust) object representation, all that's left is just a few function calls. IOW if you can account for most of the functionality you would normally have had via functions and getters by auto-generating access to naked data objects you'll realize there isn't much left. It's in that sense that it covers a good fraction of one's need for an FFI.

…

— Reply to this email directly, view it on GitHub <#38 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACSBE5SDHNAUMB3XFXZMT3XKTNPPANCNFSM6AAAAAAYKPNV3Y> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jcornaz]

Because it's specific to a language (Rust).

What do you mean? repr(c) very purpose is to interop with other languages by ensuring the same in-memory representation as what C does, so that rust can interface with any language capable of interfacing with C.

Because it is a mean of interop with other languages, it is by definition not specific to rust only.

Yes but with Python (and maybe Rust?) you can generate bindings to an
underlying C++ (or Rust?) representation.

Yes, in rust we call that repr(c)...

There's no doc. It's conceptual: if most of your API surface is shallow
bindings from say, Python to an underlying C++ (or Rust) object
representation, all that's left is just a few function calls.

If there is no doc then it is not supported. It is as simple as that.

If you somehow make it work but no documentation ever told you that things would work as they do, then you are relying on unspecified behavior and are exposing yourself to breaking changes at best, and to severe security issues at worse.

In my opinion, you should look more into repr(c), not protobuf.

[alensiljak]

Subscribing to the discussion as I've recently started playing around with Ledger-compatible implementation in Rust, and it seems that there is a slight chance of having parts of Beancount rewritten in Rust instead of C++, so I'd be interested in getting involved, eventually.

My longer-term goal would be to have a PTA in Wasm, running in a browser, which would also provide a GUI.

[alensiljak]

Here's an article that may shed some light how the interop works:

• https://maxwellrules.com/programming/rusty-python.html

[blais]

Help me out if you care. If I can see a Rust build with some native record data structure (ideally protos or capnproto but something else fine if it works well) shared across the Python boundary (with no copy), in a bazel build --- I'll very seriously consider ditching the C++ build. rules_rust works (I've tested it already, having some issues).

Rust in recent years is really taking off in the Python community and seems like a best friend to Python and a perfect fit.

[alensiljak]

https://github.com/elbaro/pyo3-prost

[alensiljak]

One issue is bugging me while researching this. I fail to see the purpose of the protobufs here. If the memory is shared directly, is there still a need to have protobuf structure? Maybe I'm missing a bigger picture.

For now, I'll look more into pyo3 as that seems to provide the best interop that is currently possible.

Also, bazel build. This is not needed for Rust but I assume there is a larger build/package system that builds other components and that the Rust build would fall into?

[blais]

Protos: You can interop with other languages seamlessly. (Unfortunately this is one of the things only those who have seen it used inside the Magic Kingdom will ever appreciate.) Think beyond just this particular use case. The parser can be leveraged to output a file of protos, some other language can manipulate it and then return it to the next stage. Objects can be unambiguously sent over the network. A text format can be used to construct them. And more. Basically in every system I work in I will always, almost invariably, attempt to represent most of the data in a language independent fashion, the advantages of that are too large.

Fully Rust: sure, but the thing is i have a lot of unit tests written in Python and it would only make sense to use those. I've made with work with C++ (the C++ code can be tested with the same Python code).

Bazel: If you want to be able to have long-term stability the ability to lock versions is necessary. For the little bits I've seen of Cargo it seems like Maven or Golang it downloads the web, and while that's convenient for prototyping it's not a serious build system for those wanting hermeticity and eventually vendoring. Needs investigation but I'm operating on the prior that everything hip these days is built for convenience and not robustness. Needs investigation?

[alensiljak]

Thanks for the explanations! I'll look deeper into these topics.
My first thought is that the interaction is already handled - that's why these platforms are interesting in the first place. For example, C (ABI) - Rust, Js - Wasm.
I'm now interested to see if/how the use of a different format opens up that scope even further.

Needs investigation?

I'd say yes! Cargo/Rust wasn't hip when it was built. The popularity came much later. I think it's been in development for over ten years.
I was under the impression that Cargo handles exactly that issue. That's the reason why crates can't be removed from the repository. One should always be sure that the dependencies will be available for download.
Anyways, it is possible to use local copies, source, etc. It's not that important. I was just wondering.

[jcornaz]

For the little bits I've seen of Cargo it seems like Maven or Golang it downloads the web, and while that's convenient for prototyping it's not a serious build system for those wanting hermeticity and eventually vendoring

To my eyes, cargo is nothing like maven. I find cargo far superior for robustness.

With cargo and crates.io:

• You can (and should) commit generated lock files that list the exact version of each and every dependency in your dependency tree. It also stores a hash for each of those dependencies and if it needs to download them, will make sure the hash matches. In other words, each build having the sameCargo.lock is guaranteed to get the exact same dependency tree.
• It will not download if the required dependencies are already present on your machine, and you may even force that behavior (I recall the flag is --offline)
• You can pull code from a git repo: beancount_parser_2 = { git = "https://github.com/jcornaz/beancount_parser_2.git", rev = "1.0.0-alpha.12" } (functionality of the lockfile still applies)
• You can use vendored code commited in your repo (or via submodule): beancount_parser_2 = { path = "./beancount_parser" }

And the points above are just the tip of the iceberg. I've used maven, gradle and npm for years, but cargo cannot be compared, it is so much better. Yes, cargo and crates.io are convenient, but it seems clear to me they've been built primarily for robustness.

[blais]

Okay so this looks interesting, but will it build other languages?

[alensiljak]

Would you mind to provide some more context, please? From what I understand, Cargo is focused on Rust and does the job well. The output is a library or an executable. It can run the tests and stuff but I don't think it is envisioned as a universal build/package/distribution system.

[jcornaz]

Yes, as @alensiljak says, cargo is specialized for rust. Even, if it does support build scripts (in the form of a build.rs file) that allow you to execute arbitrary code at compile time.

For multi-language projects, I personally use multiple tools. I use cargo for the rust part, and choose what I think is the most appropriate package manager for the non-rust parts, then I orchestrate the whole with just recipes.

But yeah, if the multi-language aspect is very important to you, I could easily believe that Bazel is better for that use case, even though I haven't tried Bazel myself yet. Or maybe the best is to use multiple tools? Hard to give a definitive answer here.