Protocol Buffers

What it is

Protocol Buffers, called protobuf for short, is Google’s language-neutral, platform-neutral, extensible way to serialize structured data. You describe your data once in a .proto file. The compiler, protoc, turns that file into code for your language. A runtime library for that language then writes and reads the data. This listing carries setup notes for protoc and for each runtime documented in the main repository, and links to the separate repositories for Go, Dart and JavaScript.

1. You describe your data in a .proto file
2. protoc turns it into code for your language
3. Your program serializes a message to bytes
4. A program in another language parses the bytes
5. Old code keeps reading data that gains new fields

Who it suits

Developers who pass structured data between services or keep it in files, and want a typed schema, compact messages and generated code. It suits teams that mix languages, such as a Java service and a Python job that share one .proto file. The overview says protobuf is most often used to define communications protocols together with gRPC, and for data storage. Skip it when a single message runs past a few megabytes, or when people need to read the data without the .proto file.

Where to use it

protoc installs on Linux, macOS and Windows, and the runtimes cover the languages below. The setup notes cover them, and link to the separate repositories for Go, Dart and JavaScript.

• protoc: a release zip for any OS, or apt, Homebrew or winget
• Python: pip install protobuf
• Java and Kotlin: Maven, with a Lite runtime for Android
• C#: the Google.Protobuf and Google.Protobuf.Tools NuGet packages
• Ruby and PHP: the google-protobuf gem, Composer and PECL
• C++: Bazel or CMake builds, or vcpkg on Windows
• Objective-C: runtime sources added to your project
• Go, Dart and JavaScript: the README sends you to their own repositories

How to set it up

Get it saves a zip of Mindket’s setup notes in your downloads folder, with a page for each runtime documented in the main repository, from the maker’s README and docs. First get protoc, then the runtime for your language.

1. Install protoc, from a release zip on GitHub or from a package manager.
2. Run protoc --version and compare it with your language’s supported version on the Version Support page, since a package manager’s protoc can be dated.
3. Install the runtime for your language.
4. Write a .proto file, then run protoc on it to generate code.

apt install -y protobuf-compiler
protoc --version
pip install protobuf

The first two lines are the installation page’s Linux route, and the third installs the Python runtime. On a Mac the page gives brew install protobuf, and on Windows winget install protobuf, each followed by protoc --version. Without a package manager, download the zip for your system from the releases page, unzip it under $HOME/.local and add the folder with protoc to your PATH. The kit shows every command.

How to use it

Write the shape of your data in a .proto file, compile it, then use the generated class. This is the overview’s example message.

edition = "2023";

message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;
}

The numbers are tags. Each field keeps its tag in the binary encoding, and the Python tutorial says tags 1 to 15 take one byte less to encode than higher ones. Compile for Python with the tutorial’s line, which names a source directory, a destination directory and the .proto path.

protoc --proto_path=$SRC_DIR --python_out=$DST_DIR $SRC_DIR/addressbook.proto

That writes addressbook_pb2.py into the destination folder. The tutorial then writes a message with SerializeToString() and reads one with ParseFromString(data). Other languages swap the option: --java_out, --kotlin_out, --csharp_out, --ruby_out and --php_out are the ones the READMEs name.

Example

You want one file that two languages read. The examples README builds an address book program for each language, and says you can create an address book with add_person_java and read it with list_people_python.

export CLASSPATH=/path/to/protobuf-java-[version].jar
make java
make python

Run add_person_java with a file name to create the address book, then list_people_python with the same file name to read it. The README shows the form ./add_person_cpp addressbook.data for C++ and says the Python scripts work the same way. You get one data file that programs in two languages share.

For newcomers and experts

If you are new. Think of a .proto file as a typed form for your data. protoc reads it and writes classes with a getter and a setter for each field, plus methods that turn the whole message into bytes and back. The overview says Protobuf is like JSON, except smaller and faster. The tutorials assume you know nothing about it, only that you can write code and handle files in your language. The Tutorials page has one for C++, C#, Dart, Go, Java, Kotlin and Python.

For experts. The overview says a deleted field’s number must never be reused, so reserve it. Old code reads new messages and ignores added fields, and new code reads old messages with defaults for what is missing, as long as you follow the update practices on protobuf.dev. The same data can serialize to many different byte strings, so two messages cannot be compared for equality without parsing them.

Python has three backends with one API. The upb backend is the default and ships in the PyPI packages, cpp is deprecated and no longer released there, and python is the pure backend. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION to choose one, and print this to see which you have. The README says this is not an officially-supported or stable API, but it is useful for ad hoc diagnostics.

from google.protobuf.internal import api_implementation
print(api_implementation.Type())

Building from source needs care. The README says builds from the head of main break now and then, and that even release branches can be unstable between release commits, so pin to a release commit. The C++ README warns that two versions of its runtime are unlikely to be ABI compatible, so recompile after an upgrade or link statically.

Good to know

The install steps are ordinary package and build steps, and a few of them reach the network.

• License: a BSD-style text with three conditions. The LICENSE file reads “Copyright 2008 Google Inc. All rights reserved.” and GitHub lists it as Other. A closing paragraph says code the compiler generates is owned by the owner of the input file, and needs a support library covered by the same license
• Network: the release-zip commands on protobuf.dev call api.github.com and download from GitHub. A CMake build fetches Abseil, Google Test and jsoncpp from GitHub when they are not installed. Cloning the repository with git downloads it from GitHub, the Bazel WORKSPACE route’s http_archive downloads from its url, and go install fetches the Go plugin. Package managers reach their own registries
• Versions: pre-built binaries exist only for released versions. For the head of main you build protoc yourself, and the README points to the Maven repository for old ones
• Fit: the overview says protobuf is not a formal standard of any organization, which makes it unsuitable where legal or other requirements call for building on standards. It does not compress messages, and it suits scientific work with large arrays of floating point numbers less well
• Support: Google open sourced it in 2008. The overview says it accepts select pull requests but cannot always prioritize requests that do not fit its own needs
• Notes: the setup notes are Mindket’s, and Google has not made or endorsed them
• Removal: the READMEs give no uninstall steps. Remove each part with the tool you installed it with, or delete the unzipped release files and the PATH line