Skip to content

v1.20.0
Compare
Choose a tag to compare
@dsnet dsnet released this 02 Mar 20:21
· 327 commits to master since this release
v1.20.0
aa81c06

Overview

The google.golang.org/protobuf module is a major revision of the Go bindings for protocol buffers. The API has been completely overhauled with a focus on simplicity and clarity. Deprecated types and functions are removed, complicated packages are split into more manageable pieces, and internal details are separated from the public API. Particular attention was made to ensure that this implementation complies with the protocol buffer language as closely as possible to ensure proper interoperability between the Go implementation and implementations for other languages.

The first version of this module is v1.20.0. The version number is deliberately chosen to avoid overlap with existing and anticipated versions of the github.com/golang/protobuf module. This enables the version alone to unambiguously distinguish between the two modules.

The minimally supported version of Go is v1.9.

Backwards compatibility

The API in this module is not drop-in compatible with the earlier github.com/golang/protobuf module. However, the new API does strive when sensible to be compatible for easier migration.

Version v1.4.0 of the github.com/golang/protobuf module implements most of the existing API in terms of this module, which provides a number of benefits:

  • Shared type registry: It ensures that the two modules share a unified type registry, which is necessary for dynamic usages of protocol buffers to work properly.
  • Automatic improvements: It provides the older module with a degree of automatic upgrading whenever bug fixes and improvements are made to the newer module.
  • Long-term maintainability: It keeps the older module more maintainable since it is largely a thin shim over the newer module.

Some existing features present in the github.com/golang/protobuf module are currently unavailable in this module:

  • Manipulation of the wire format: The older module provides functionality for directly manipulating the raw wire format through methods on the proto.Buffer type.
  • Helper functions for well-known types: The older module provides the ptypes package containing helper functions for interacting with the well-known types such as Any or Timestamp messages.

We plan to add these features to this module in the near future and improve upon them.

Package index

Summary of the packages provided by this module:

  • proto: Package proto provides functions operating on protobuf messages such as cloning, merging, and checking equality, as well as binary serialization.
  • encoding/protojson: Package protojson serializes protobuf messages as JSON.
  • encoding/prototext: Package prototext serializes protobuf messages as the text format.
  • reflect/protoreflect: Package protoreflect provides interfaces to dynamically manipulate protobuf messages.
  • reflect/protoregistry: Package protoregistry provides data structures to register and lookup protobuf descriptor types.
  • reflect/protodesc: Package protodesc provides functionality for converting descriptorpb.FileDescriptorProto messages to/from the reflective protoreflect.FileDescriptor.
  • testing/protocmp: Package protocmp provides protobuf specific options for the cmp package.
  • testing/prototest: Package prototest exercises the protobuf reflection implementation for concrete message types.
  • types/dynamicpb: Package dynamicpb creates protobuf messages at runtime from protobuf descriptors.
  • types/known/anypb: Package anypb is the generated package for google/protobuf/any.proto.
  • types/known/emptypb: Package emptypb is the generated package for google/protobuf/empty.proto.
  • types/known/timestamppb: Package timestamppb is the generated package for google/protobuf/timestamp.proto.
  • types/known/durationpb: Package durationpb is the generated package for google/protobuf/duration.proto.
  • types/known/wrapperspb: Package wrapperspb is the generated package for google/protobuf/wrappers.proto.
  • types/known/structpb: Package structpb is the generated package for google/protobuf/struct.proto.
  • types/descriptorpb: Package descriptorpb is the generated package for google/protobuf/descriptor.proto.
  • types/pluginpb: Package pluginpb is the generated package for google/protobuf/compiler/plugin.proto.
  • compiler/protogen: Package protogen provides support for writing protoc plugins.
  • cmd/protoc-gen-go: The protoc-gen-go binary is a protoc plugin to generate a Go protocol buffer package.

Notable changes

This section summarizes notable changes in this release relative to v1.3.4 of the github.com/golang/protobuf module.

Message type

The definition of the proto.Message interface type has changed (see v1 Message and v2 Message). The new type definition contains a behaviorally-complete specification of a protobuf message. All functions in this module that operate on a proto.Message can be used with values of any concrete type which properly implements this interface.

The github.com/golang/protobuf/proto package contains functions to convert between the old and new proto.Message types (see MessageV1 and MessageV2). These functions are guaranteed to work with types generated by protoc-gen-go, however there is no guarantee that it handles hand-crafted messages or messages generated by different generators.

Reflection

A key feature of the new Go protobuf API is support for protobuf reflection, which is the ability to programmatically interact with a message at runtime. The ProtoReflect method implemented by every proto.Message returns a reflective view of that message. Using protobuf reflection instead of Go reflection is generally easier since it closely follows the protobuf data model, and also more maintainable since it relies on a stable API that is agnostic to the exact underlying implementation of a message. On the other hand, using Go reflection on protobuf messages often needs to make low-level assumptions about a particular implementation of protobuf messages that may not be stable (e.g., assuming that the Go struct field ordering exactly matches the proto message field ordering).

See the protoreflect package for more details.

Wire serialization

The API for wire serialization has changed slightly for better usability. Optional arguments for controlling serialization are specified through the proto.MarshalOptions and proto.UnmarshalOptions types. Both option types have an AllowPartial option to specify that the operation should avoid checking required fields for initialization.

JSON serialization

The JSON implementation is improved in the areas of correctness, usability, and performance.

Summary of changes:

  • Top-level functions (see protojson.Marshal and protojson.Unmarshal) operate on []byte instead of io.Reader and io.Writer to match the other serialization functions in the module.
  • There is no API for parsing JSON in a streaming manner. Instead, use the standard json.Decoder.Decode method to unmarshal JSON tokens into a json.RawMessage in a streaming manner. The raw JSON can then be passed to protojson.Unmarshal.
  • There is no special support for types implementing the older jsonpb.JSONPBMarshaler and jsonpb.JSONPBUnmarshaler interfaces. Instead, custom message types should implement the new proto.Message interface.
  • Marshal and unmarshal options have been renamed:
    • OrigName → UseProtoNames
    • EnumAsInts → UseEnumNumbers
    • EmitDefaults → EmitUnpopulated
    • AllowUnknownFields → DiscardUnknown
    • AnyResolver → Resolver
  • protojson.Marshal and protojson.Unmarshal both support an AllowPartial option which specifies that the operation should avoid checking required fields. The default behavior is to report an error for any uninitialized required fields.
  • protojson.Marshal and protojson.Unmarshal both provide proper support for the FieldMask well-known type.
  • protojson.Unmarshal resets the message before performing the operation, dropping any form of merge semantics. This is done for the new module since the protocol buffer language specifies no behavior for how merging in JSON ought to operate.
  • protojson.Marshal returns an error when serializing a string field containing invalid UTF-8. Previously, it would silently mangle invalid UTF-8 in the string with the Unicode replacement character.
  • protojson.Unmarshal returns an error when deserializing input string fields containing invalid UTF-8, while previously it would silently deserialize the string into the field.
  • protojson.Unmarshal returns an error when parsing an input with duplicate entries in a JSON object. Previously, it would accept the last entry listed syntactically. This affects parsing of JSON into a protobuf message and into a map field.

The output of the serializer is deliberately unstable to prevent users from accidentally depending on the output as being stable (when it isn’t). This ensures that we continue to freely change the output to fix bugs and make improvements.

The changes above (although correct according to protobuf semantics) unfortunately breaks a sufficient amount of user code that unfortunately depends either on buggy behavior or assumes that the output will remain the same forever. The implementation of JSON serialization in the older module will not wrap this module’s implementation. See that module’s release notes for details.

Text serialization

The text format implementation is improved in the areas of correctness, usability, and performance.

Summary of changes:

  • Text serialization is separated from the proto package and placed in its own prototext package.
  • Top-level functions (see prototext.Marshal and prototext.Unmarshal) operate on []byte instead of io.Writer and string to match the other serialization functions in the module.
  • There is no special support for types implementing the encoding.TextMarshaler and encoding.TextUnmarshaler interfaces. Instead, custom message types should implement the new proto.Message interface.
  • prototext.Marshal and prototext.Unmarshal both support an AllowPartial option which specifies that the operation should avoid checking required fields. The default behavior is to report an error for any uninitialized required fields.
  • prototext.Marshal returns an error when serializing a string field containing invalid UTF-8.
  • prototext.Marshal output changed in compatible ways:
    • By default, unknown fields are ignored. To explicitly serialize them, specify the EmitUnknown option.
    • By default, the Any well-known type is expanded with the textual representation of the underlying message value. To disable this behavior, pass an empty protoregistry.Types registry as the Resolver option.
    • Protobuf string values are serialized as UTF-8 instead of strict ASCII. Previously, it would escape valid Unicode characters to ensure that the output was ASCII, which made strings containing non-English text unreadable.
    • Protobuf bytes values are serialized as a string using hexadecimal escaping instead of octal escaping.
    • The delimiter surrounding messages is switched to {} characters to be more consistent with the implementation in other languages. Previously, it used <> characters.
    • The field separator : is always emitted between the field name and the field value, even though the separator is optional for message fields.
  • prototext.Unmarshal provides the DiscardUnknown option to ignore unknown fields when parsing input. By default, it rejects unknown fields as an error.

The output of the serializer is deliberately unstable to prevent users from accidentally depending on the output as being stable (when it isn’t). This ensures that we continue to freely change the output to fix bugs and make improvements.

The changes above (although correct according to protobuf semantics) unfortunately breaks a sufficient amount of user code that unfortunately depends either on buggy behavior or assumes that the output will remain the same forever. The implementation of text serialization in the older module will not wrap this module’s implementation. See that module’s release notes for details.

Extensions

Extension fields (a proto2 feature) are eagerly parsed while unmarshaling a message. Previously, the raw bytes for extensions were simply stored in the message and later lazily parsed during a proto.GetExtension call. The change in behavior is to bring the Go implementation in closer compliance with all the other major language implementations of protocol buffers. A consequence of this change is that extension fields with invalid data will cause the initial unmarshal to fail, rather than a later proto.GetExtension access.

The API for accessing extension fields is simplified:

  • proto.GetExtension no longer returns an error. Since extensions are eagerly unmarshaled, there is no lazy parse operation that can fail.
  • proto.GetExtension no longer returns raw field data for unknown extensions when given a type-incomplete extension type. Users should instead use the protoreflect.Message.GetUnknown method to access unknown extension fields.
  • proto.SetExtension no longer returns an error. Passing in the wrong type for the field value results in a panic.
  • Scalar extension fields (except bytes) are now represented by non-pointer Go types (i.e., bool, int32, int64, uint32, uint64, float32, float64, string, and named enum types). Previously, they were represented by a pointer to those types (e.g., *bool, *float32, or *foopb.MyEnum).

Passing in an extension type to any of the extension functions that does not extend the given message results in a panic.

Nil values

The generated API is not an exact mapping to the protobuf data model, but rather a superset of the latter. This implies that there are certain representations of values in Go that are not observable through protobuf reflection and are not preserved when round-tripping through the wire format (or any other format).

The following is a non-exhaustive list of ways where differing representations of concrete values in Go has no distinguishable difference according to the protobuf data model.

  • Nil pointers for messages: Protobuf messages have a concept of presence for singular message fields in another message. However, there is no concept of presence for messages in other contexts (e.g., messages in a list or map). Succinctly, presence is not a first-class property of the protobuf message type, but rather a property of the field within a message.

    In the generated Go API, we use a nil pointer to represent presence for singular message fields (that are not part of a oneof). The ability to have a typed nil pointer to a message (e.g., (*foopb.MyMessage)(nil)) outside the context of a singular message field opens the questions of what this means from protobuf’s perspective. To ensure a consistent meaning within the module, we treat a typed nil pointer as being equivalent to a read-only empty message of that type. Thus, it can be passed to functions that only read the message (e.g., proto.Marshal), but will panic when passed to functions that mutate the message (e.g., proto.Unmarshal or proto.Reset).

    This is consistent with the behavior of the message when accessor methods (e.g., m.GetFoo) are called on a typed nil message, where they functionally present the message as if it were empty.

  • Nil or empty slices for repeated fields: There is no concept of presence for repeated fields in protobuf. Thus, []T(nil) and []T{} are both treated identically as an empty protobuf list. When unmarshaling or merging, a nil slice is used to store an empty protobuf list.

  • Nil values in slices for repeated fields: There is no concept of presence for individual elements within a protobuf repeated field. However, some element types for a Go slice can be set to nil (e.g., byte slices or pointers to message types). From the protobuf perspective, these are treated identically as an empty bytes value or empty message. When unmarshaling or merging, an allocated but empty value is used for bytes values (e.g., []byte{}) and message values (e.g., new(foopb.MyMessage)) in a protobuf list.

  • Nil or empty maps for map fields: There is no concept of presence for map fields in protobuf. Thus, map[K]V(nil) and map[K]V{} are both treated identically as an empty protobuf map. When unmarshaling or merging, a nil map is used to store an empty protobuf map.

  • Nil values in maps for map fields: There is no concept of presence for individual entry values within a protobuf map field. However, some value types for a Go map can be set to nil (e.g., byte slices or pointers to message types). From the protobuf perspective, these are treated identically as an empty bytes value or empty message. When unmarshaling or merging, an allocated but empty value is used for bytes values (e.g., []byte{}) and message values (e.g., new(foopb.MyMessage)) in a protobuf map.

  • Nil or empty slice for singular bytes in proto3: There is no concept of presence for singular bytes field in the proto3 syntax. Thus, []byte(nil) and []byte{} are both treated identically as an unpopulated protobuf bytes field. When unmarshaling or merging, a nil slice is used to store any empty protobuf bytes value.

  • Nil pointers to wrapper types for oneof field: Protobuf fields within a oneof have presence. The generated Go represents oneofs as an interface, where a given field is populated by allocating a wrapper struct with the field value set (e.g., m.Oneof = &foopb.Message_Field{"hello"}). The Go API unfortunately allows for two-levels of presence: 1) for whether the interface itself is nil, and 2) for whether the field wrapper struct itself is nil. Since the second layer cannot be represented in the protobuf data model, we treat a nil wrapper type (e.g., (*foopb.Message_Field)(nil)) as being identical to the entire oneof being unpopulated. When unmarshaling or merging, a nil interface value is used to represent an unpopulated oneof.

Previous versions of the github.com/golang/protobuf module had inconsistent behavior with regard to how the cases above were handled. Some functions would ignore the difference, others would preserve it, while others would panic. The new module aims to provide uniform behavior consistent with the protobuf data model.

Errors

Error messages may have changed in trivial ways, which may cause brittle tests checking the exact text of errors returned from the protobuf module to fail. The principled way to test for errors returned from this module is to check whether errors.Is reports a match with proto.Error.

Generated code

Code generated by the v1.20 and higher protoc-gen-go compiler plugin requires the google.golang.org/protobuf module. On the other hand, the runtime provided by this module supports generated code produced by v1.0 and higher of the older protoc-gen-go compiler plugin. Users should strive to use the same version for both the runtime library and the protoc-gen-go plugin used to generate the Go bindings.

Generated enums have additional Descriptor, Type, and Number methods that provide a reflective view over the enum value (see protoreflect.Enum). Generated messages have an additional ProtoReflect method that provides a reflective view over the message value (see protoreflect.ProtoMessage).

The String method for generated messages produces different output. There are no guarantees for the output of String other than it being a humanly readable representation of the message. Brittle code that depends on the String output being stable should use appropriate protobuf functionality (e.g. proto.Equal) instead of relying on direct string comparison. If a use-case cannot be migrated, the old output can be reproduced using the proto.CompactTextString function from the older module.

All XXX prefixed fields (e.g., XXX_NoUnkeyedLiteral, XXX_unrecognized, or XXX_sizecache) and methods (e.g., XXX_Size, XXX_Marshal, XXX_Unmarshal, XXX_Merge, XXX_DiscardUnknown, XXX_InternalExtensions, XXX_OneofWrappers, or XXX_WellKnownType) have been removed and are instead implemented using unexported fields in the generated struct for a protobuf message. Any code directly or indirectly depending on these fields or methods need to be migrated to use principled APIs that provide equivalent and stable functionality. The compatibility document for Go protobufs reserves the right to add, modify, or remove these declarations.

The presence of internal fields in a generated message may cause existing usages of reflect.DeepEqual to behave differently. This occurs because reflect.DeepEqual blindly compares unexported fields without understanding the semantic significance of any of these fields. As a result, changes to internal implementation details are unfortunately observable to the user. The principled way to compare protobuf messages is to use proto.Equal (if the values being compared are messages) or to use cmp.Equal with the cmp.Comparer(proto.Equal) option specified. For more complex comparison use-cases, see the protocmp package.

To access the unknown fields of a message m (previously available through the undocumented XXX_unrecognized field), use m.ProtoReflect().GetUnknown().

gRPC support

The v1.20 protoc-gen-go does not support generating gRPC service definitions. In the future, gRPC service generation will be supported by a new protoc-gen-go-grpc plugin provided by the Go gRPC project.

The github.com/golang/protobuf version of protoc-gen-go continues to support gRPC and will continue to do so for the foreseeable future.

Performance

Some performance benefits are observed due to cleanups that occurred while porting the older runtime implementation over to this module. It is our goal that this module’s implementation maintains performance parity with the older module’s implementation at minimum. If you experience measurable performance degradation, please file a bug with a reproduction.

Indirect use of this module through the older github.com/golang/protobuf module may experience a slight performance regression when operating on messages that have not been regenerated using the protoc-gen-go binary provided in this module. This degradation occurs because the older module may need to wrap an older message to implement the new proto.Message interface in order to call the new module’s APIs. We strongly recommend regenerating messages using the latest protoc-gen-go plugin before making any judgements regarding performance.

Upcoming breaking changes

Per the compatibility agreement for Go protobufs, we promise to make announcements about deliberate or potentially breaking changes 6 months in advance.

None of these changes are effective yet in this release.

Enforcement of namespace uniqueness

A future release of Go protobufs will strictly adhere to protocol buffer language semantics and panic loudly at program init whenever conflicting declarations are detected during registration. In this release, the registry will log every conflict detected. For example:

WARNING: message my.proto.package.MyMessage is already registered
    previously from: "some/path/to/generated/go_proto_package"
    currently from:  "other/path/to/generated/go_proto_package"

See the developer guide for more information on how to resolve namespace conflicts.

Required go_package option

Source .proto files may contain a go_package option specifying the import path of the Go package containing the generated code for the file. A future version of protoc-gen-go will require that the import path of every generated file be explicitly specified with either a go_package option or as a command-line flag. The purpose of this change is to foster a world where there is one canonical Go package that represents every .proto file. This avoids type variance issues at scale where dependencies rely on disjoint Go types that functionally represent the same protobuf message.

See the developer guide for more information on the go_package option.

Deprecation of protoc-gen-go flags

The plugins, import_prefix, and import_path flags for protoc-gen-go are deprecated.

The plugins flag provided a way to invoke the gRPC code generator. The gRPC code generator is being split into a separate protoc-gen-go-grpc plugin binary.

The import_prefix flag prepends prefix to the import paths of packages in generated code. The precise behavior of this flag was surprising and rarely useful.

The import_path flag provided the ability to specify the expected Go import path for the current .proto file being compiled. Instead, the go_package option should be specified in the .proto file.

Deprecation of generated APIs

The ExtensionRangeArray method on generated messages is deprecated and will be removed in the future. To find the extension ranges of a message m, use m.ProtoReflect().Descriptor().ExtensionRanges().

The Descriptor method of generated messages and the EnumDescriptor method of generated enums are deprecated and may be removed in the future. To find the descriptor of a message m, use m.ProtoReflect().Descriptor(). To find the descriptor of an enum e, use e.Descriptor(). To convert a protoreflect descriptor to a descriptor message, use the
protodesc package.

The UnmarshalJSON method of generated proto2 enums is deprecated and may be removed in the future.

The generated protobuf and json tags on struct fields are deprecated and may be removed in the future. The protobuf tags have been superseded by native support for protobuf reflection, while the json tags have given the unfortunate illusion that generated messages work correctly with the encoding/json package when they actually do not. The protojson package should be used instead for JSON serialization.

Assets 8