ProtoBuf.jl
provides a compiler and codec for Google's Protocol Buffers serialization format.
Use the protojl
function to translate your .proto
files to Julia, then you can encode and decode your messages with
encode(e::ProtoEncoder, x::YourMessage)
+decode(d::ProtoDecoder, ::Type{YourMessage})
Where the ProtoEncoder
and ProtoDecoder
are simple types wrapping your IO
.
The package is not currently registered, to install it, use:
import Pkg; Pkg.add("ProtoBuf")
Given a test.proto
file in your current working directory:
syntax = "proto3";
+
+enum MyEnum {
+ DEFAULT = 0;
+ OTHER = 1;
+}
+
+message MyMessage {
+ sint32 a = 1;
+ repeated string b = 2;
+}
You can generate Julia bindings with the protojl
function:
julia> using ProtoBuf
+
+julia> protojl("test.proto", ".", "output_dir")
This will create a Julia file at output_dir/test_pb.jl
which you can simply include
and start using it to encode and decode messages:
julia> include("output_dir/test_pb.jl")
+Main.test_pb
+
+julia> io = IOBuffer();
+
+julia> e = ProtoEncoder(io);
+
+julia> encode(e, test_pb.MyMessage(-1, ["a", "b"]))
+8
+
+julia> seekstart(io);
+
+julia> d = ProtoDecoder(io);
+
+julia> decode(d, test_pb.MyMessage)
+Main.test_pb.MyMessage(-1, ["a", "b"])
If you are curious, this is what the generated file looks like:
# Autogenerated using ProtoBuf.jl v0.1.0 on 2022-07-25T11:32:05.368
+# original file: /Users/tdrvostep/_proj/ProtoBuf.jl/test.proto (proto3 syntax)
+
+module test_pb
+
+import ProtoBuf as PB
+using ProtoBuf: OneOf
+using EnumX: @enumx
+
+export MyEnum, MyMessage
+
+@enumx MyEnum DEFAULT=0 OTHER=1
+
+struct MyMessage
+ a::Int32
+ b::Vector{String}
+end
+PB.default_values(::Type{MyMessage}) = (;a = zero(Int32), b = Vector{String}())
+PB.field_numbers(::Type{MyMessage}) = (;a = 1, b = 2)
+
+function PB.decode(d::PB.AbstractProtoDecoder, ::Type{<:MyMessage})
+ a = zero(Int32)
+ b = PB.BufferedVector{String}()
+ while !PB.message_done(d)
+ field_number, wire_type = PB.decode_tag(d)
+ if field_number == 1
+ a = PB.decode(d, Int32, Val{:zigzag})
+ elseif field_number == 2
+ PB.decode!(d, b)
+ else
+ PB.skip(d, wire_type)
+ end
+ end
+ return MyMessage(a, b[])
+end
+
+function PB.encode(e::PB.AbstractProtoEncoder, x::MyMessage)
+ initpos = position(e.io)
+ x.a != zero(Int32) && PB.encode(e, 1, x.a, Val{:zigzag})
+ !isempty(x.b) && PB.encode(e, 2, x.b)
+ return position(e.io) - initpos
+end
+function PB._encoded_size(x::MyMessage)
+ encoded_size = 0
+ x.a != zero(Int32) && (encoded_size += PB._encoded_size(x.a, 1, Val{:zigzag}))
+ !isempty(x.b) && (encoded_size += PB._encoded_size(x.b, 2))
+ return encoded_size
+end
+end # module
Below is a list of notable differences that were introduced in 1.0.
For translating proto files, use protojl
function (previously protoc
). To decode proto messages, use the decode
method (previously readproto
) and to encode Julia structs, use encode
(previously writeproto
). See Quickstart for an example.
Messages are now translated to immutable structs. This means that code that used the mutable structs to accumulate data will now have to prepare each field and construct the struct after all of them are ready.
By default, the generated structs don't share any common abstract type (well, except Any
), when you set the common_abstract_type
option to true
, every struct definition will be a subtype of ProtoBuf.AbstractProtoBufMessage
.
The naming of nested messages and enums now generates names, that cannot collide with other definitions. For example:
message A {
+ message B {}
+}
now generates structs named A
and var"A.B"
. In protobuf, it is legal to a define message called A_B
but not A.B
, which is a syntax to refer to these nested definitions.
Similarly, field names that coincide with Julia reserved keywords were previously prefixed with an underscore (e.g. _function
), now we prefix them with a #
(e.g. var"#function"
).
EnumX.jl is used to define enums
instead of NamedTuple
s. This means that to get the type of the enum, one must use MyEnum.T
as MyEnum
is a Module
. You can now dispatch on Base.Enum
when working with @enumx
-based enums.
oneof
fields are now explicitly represented in the generated struct. Specifically, for a message
message MyMessage {
+ oneof oneof_field {
+ int32 option1 = 1;
+ string option2 = 2;
+ }
+}
a struct with a single field will be generated:
struct MyMessage
+ oneof_field::Union{Nothing,OneOf{<:Union{Int32,String}}}
+end
Once instantiated, it will contain a value like OneOf{:option1, 42}
or OneOf{:option2, "The answer to life, the universe, and everything"}
. One can access the name and the value of the field via the :name
and :value
properties, respectively. Dereferencing the field will return the value of the field (e.g. my_message.oneof_field[] == 42
)
For .proto
files that are packages, a nested directory structure will be generated. For example, {file_name}.proto
containing package foo_bar.baz_grok
, the following directory structure is created:
root # `output_directory` arg from from `protojl`
+└── foo_bar
+ ├── foo_bar.jl # defines module `foo_bar`, imports `baz_grok`
+ └── baz_grok
+ ├── {file_name}_pb.jl
+ └── baz_grok.jl # defines module `baz_grok`, includes `{file_name}_pb.jl`
You should include the top-level module of a generated package, i.e. foo_bar.jl
in this example. When reading .proto
files that use import
statements, the imported files have to be located at the respective import paths relative to search_directories
.
.proto
files that don't have a package
specifier will generate a single file containing a module. For example, {file_name}.proto
will translate to a {file_name}_pb.jl
file defining a {file_name}_pb
module. You can generate a file without a module by providing always_use_modules=false
options to protojl
.
By default, there are no additional constructors generated for the Julia structs. You can use the add_kwarg_contructors=true
option to protojl
to create outer constructors that accept keyword arguments and provide default values where available.