c++ impl 100x+ slower than golang impl

[githubaccount888]

The C++ jsonnet implementation is orders of magnitude slower than the golang implementation. I hit the issue using std.sort() but the issue may very well be broader than that. Both were run with v0.16.0. The attached example uses a 10000 element list but the issue is still obvious down with lengths around ~1000. Based on my testing it appears the C++ version also scales worse than golang as the problem size increases.

test.txt

C++

$ time $(./jsonnet test.txt &> /dev/null)
real	0m18.887s
user	0m18.800s
sys	0m0.085s

golang

$ time $(~/.go/bin/jsonnet test.txt &> /dev/null)
real	0m0.079s
user	0m0.095s
sys	0m0.017s

As far as I can tell C++ is still the recommended "reference" implementation, so it's quite surprising to see it perform so poorly. I originally hit the problem evaluating jsonnet through Python bindings ("jsonnet" module) -- there is an analogous gojsonnet module but that requires installing the entire golang toolchain, which is an absurdly heavy dependency just for evaluating jsonnet.

I know nothing about the jsonnet codebase, but just poking around for 5 seconds it appears they supposedly share the same std library implementation (which is written in jsonnet), so at first glance this behavior was even more surprising. Digging slightly deeper, however, it looks like the golang implementation ends up overriding certain functions in builtins.go (presumably to fix performance issues). If that's true, it seems like the same steps should have also been taken in the true "recommended" C++ implementation (or better yet just fix in the std library itself and remove the implementation-specific behavior).

[sbarzowski]

As far as I can tell C++ is still the recommended "reference" implementation, so it's quite surprising to see it perform so poorly.

Actually not. Maybe we should be more clear in our messaging, but the Go version is a newer implementation, maintained by the same people and it is supposed to eventually completely replace this one. Most of our effort is concentrated there.

One of the things that we were able to improve in the Go version is the ease of writing native implementation of builtins. And the advantage of builtins is most pronounced for the simplest operations on lots of small values such as your sorting example (which is admittedly ridiculously slow in cpp-jsonnet).

The C++ implementation will be supported, in the sense of implementing any new additions to the language and bug fixes for the foreseeable future. However, if you care about getting the best performance I definitely recommend switching to go-jsonnet. C and Python bindings are available, although the Python support is very new, so there may be some rough edges there. The golang toolchain is only needed to compile the Python extension – you can avoid that by building a wheel once, and then distributing a binary version to any machines you want it installed on. (I hope we will provide pre-built wheels of the gojsonnet Python package, but I haven't figured out how to set it up yet.)

[githubaccount888]

Thanks for the info... yeah I agree the messaging could probably be improved here.

On the main go-jsonnet Github, it says

This an implementation of Jsonnet in pure Go. It is feature complete but is not as heavily exercised as the Jsonnet C++ implementation. Please try it out and give feedback.

That still suggests to me the Go version is still somewhat experimental and to be considered 'second-class' next to the C++ version. I thought I read a more explicit sentiment elsewhere but I can't find it anymore (perhaps was 1-2 years ago and that phrasing has since changed).

Also maybe it would make sense to just make the known C++ performance issues front and center so people don't spend time going down the same rabbit hole? I'm guessing the motivation for maintaining two separate implementations is primarily testing and flushing out bugs? If that is the case, maybe highlight that the C++ has essentially been deprecated to a "gold reference" implementation where performance isn't a primary concern?

Otherwise, to end users without this background knowledge, it will continue being a huge surprise when the C++ struggles on basic tasks like sorting 1,000 items.

[brettviren]

I am very interested to learn there are C bindings to the faster Go version!

Do the "todo()" calls I find here:

https://github.com/google/go-jsonnet/blob/master/c-bindings/libjsonnet.cpp#L95

indicate the C bindings are still a work in progress? Or, am I looking in the wrong place?

Thanks!

[sbarzowski]

These 4 functions are missing, so it's not 100% complete. These are convenience functions, so they were not prioritized. Everything else should work fine.

There is nothing technically difficult about implementing them. If it's blocking you from adopting go-jsonnet, please open an issue to let us know (PRs also welcome ofc). It's basically a matter of implementing variants of this https://github.com/google/go-jsonnet/blob/master/c-bindings/c-bindings.go#L140 and this https://github.com/google/go-jsonnet/blob/master/c-bindings/c-bindings.go#L162 using https://github.com/google/go-jsonnet/blob/master/vm.go#L206 and this https://github.com/google/go-jsonnet/blob/master/vm.go#L219.

[brettviren]

+1 thanks.

I have no rush on my end. I don't know Go so there's some bar for me there. But I think simply knowing this alternative is possible is a relief. We often hit Jsonnet-C++ compile speed issues which so far can be worked around by taking care of what we call deeply in some nested structure. I expect next time I hit a similar slow down will motivate me to look more closely at using the C++ binding to go-jsonnet.

[He-Pin]

https://github.com/databricks/sjsonnet has a scala-native implementation.