v1.27.1

Notable changes since v1.27.0:

• CL/331149: cmd/protoc-gen-go: fix generation of enum defaults

v1.27.0

• Overview
• Notable changes
  • Reflectively ranging over a message
• Upcoming breakage changes

Overview

The release provides new functionality for iterating through a message using protobuf reflection. There are some minor changes to the code generator.

Notable changes

New features:

• CL/309669: testing/protopack: add Message.UnmarshalAbductive

Bug fixes:

• CL/317430: encoding/prototext: fix skipping of unknown fields
• CL/321529: internal/impl: support typed nil source for Merge of aberrant messages

Generator changes

• CL/305574: cmd/protoc-gen-go: remove generation of the ExtensionRangeArray method
• CL/319649: cmd/protoc-gen-go: avoid referencing remote enum values by name
• CL/316949: compiler/protogen: relax rules for valid import paths
• CL/306209: cmd/protoc-gen-go: add protoc suffix

Reflectively ranging over a message

• CL/236540: reflect: add protopath and protorange packages

The new reflect/protorange package supports recursively ranging through all populated fields of a message. There are many use cases for such a feature. See the examples for inspiration.

Upcoming breakage changes

This release removes generation of the ExtensionRangeArray method, as originally announced since the v1.20.0 release on March 2nd, 2020. Our analysis of the entire public module proxy found no static usages of this method. This method is pseudo-internal to the implementation and we expect removal of it to have no material impact. If something is broken by this change, please file an issue and we can consider re-generating this method in a patch release.

There are no new upcoming breaking changes to announce in this release.

v1.26.0

• Overview
• Notable changes
  • Reduced dependencies
  • Require Go import path
  • Fatal namespace conflicts
  • Support for aberrant message types
• Upcoming breakage changes

Overview

This release reduces dependencies on other modules, enforces strict behavior with generated Go packages and name conflicts, expands support for aberrant message types, and provides minor new features in protobuf reflection.

Notable changes

New features:

• CL/242377: proto: add MessageName helper
• CL/236777: reflect/protoreflect: add MessageFieldTypes
• CL/239838: reflect/protoreflect: add FieldDescriptor.TextName
• CL/238000: reflect/protoreflect: improve source information usability

Behavior changes:

• CL/240017: internal/impl: introduce instability to protoreflect.Message.Range order
• CL/262681: internal/encoding/text: escape \x7f (DEL) in strings
• CL/262682: internal/encoding/text: escape Unicode control characters in strings
• CL/259900: internal/filedesc: remove ProtoLegacyRawDesc method
• CL/286132: internal/descfmt: always include type name in FormatList
• CL/302330: cmd/protoc-gen-go: support --help flag
• CL/301952: cmd/protoc-gen-go: print version to stdout
• CL/244297: all: return less-specific, but more informative wire unmarshal errors

Bug fixes:

• CL/247458: testing/protocmp: fix representation of empty bytes
• CL/241658: testing/prototest: fix suggestion for field name
• CL/259899: reflect/protodesc: fix round-tripping for package field
• CL/247737: encoding/protojson: restrict valid values for google.protobuf.Value.number_value
• CL/246097: types/known/fieldmaskpb: repeated and map fields are only valid in the last position of a path
• CL/244718: internal/impl: make errInvalidUTF8 be a proto.Error

Performance optimizations:

• CL/253100: proto/equal: reduce equal scalar value allocation
• CL/240378: reflect/protoregistry: avoid checking for '/' in FindMessageByName

Reduced dependencies

• CL/259901: cmd/protoc-gen-go: remove reference to legacy ProtoPackageIsVersion4
• CL/262679: all: rely on dynamic dependency check for genproto
• CL/262683: all: remove weak dependency on github.com/golang/protobuf
• CL/262684: all: add weak dependency on github.com/golang/protobuf
• CL/235283: all: update protobuf toolchain dependency

Go code generated by protoc-gen-go no longer have a hard dependency on the github.com/golang/protobuf module. In previous releases, the generator would emit a reference to the ProtoPackageIsVersion4 constant to statically enforce that a sufficiently new version of the older module was being linked in. In the last year, most of the Go ecosystem has moved to Go modules, where we can rely on Go modules to enforce a minimum version constraint on the older protobuf module. While there continues to be a cyclic dependency between the github.com/golang/protobuf and google.golang.org/protobuf modules, the cyclic dependency was briefly broken and re-added to clear out the infinitely growing list of go.sum entries.

The module dependency on the google.golang.org/genproto module has been removed. Any program that depends on both google.golang.org/protobuf and google.golang.org/genproto must use at least version cb27e3aa from May 26th, 2020. There is a runtime check at program initialization that panics when too old of a version of the genproto module is being linked into the binary.

The well-known types provided by this module are built from v3.15.3 of the protobuf toolchain.

Require Go import path

• CL/301953: compiler/protogen: require that the import path be specified

As mentioned in the v1.20.0 release from March 2nd, 2020, the Go generator will require that the Go import path be specified for all .proto files being generated and their transitive dependencies. Since v1.20.0, every occurrence of a .proto file missing a Go import path has resulted in a warning being printed to standard error that this will eventually become a fatal error. The Go import path is normally specified by declaring a go_package option in the .proto file. Alternatively, this can be specified at compile-time with a M flag passed on the command line. See the developer documentation for more information on the go_package option and M flag.

Fatal namespace conflicts

• CL/235298: reflect/protoregistry: panic upon registration conflicts
• CL/295349: reflect/protoregistry: add compile-time opt-out for registration conflicts

As mentioned in the v1.20.0 release from March 2nd, 2020, the Go protobuf runtime will fatally fail if it detects the registration of duplicate names. Since v1.20.0, every occurrence of a conflict has resulted in a warning being printed to standard error that this will eventually become a fatal error. See the developer documentation for more information about what a namespace conflict is and how to resolve it.

While it is preferable that the source of the conflict be fixed, the fatal error can be immediately worked around in one of two ways:

1. At compile time. The default behavior for handling conflicts can be specified at compile time with a linker-initialized variable:

go build -ldflags "-X google.golang.org/protobuf/reflect/protoregistry.conflictPolicy=warn"

2. At program execution. The behavior for handling conflicts when executing a particular Go binary can be set with an environment variable:

GOLANG_PROTOBUF_REGISTRATION_CONFLICT=warn ./main

This will cause the runtime to use the conflict handling behavior from v1.25.0 and earlier, which is to simply warn about them.

Support for aberrant message types

• CL/300869: internal/impl: add runtime support for aberrant messages
• CL/244937: internal/impl: add runtime support for *[]byte unknown representation

This project's compatibility document specifies that we only guarantee runtime compatibility with messages generated by protoc-gen-go. Use of this module's runtime with any other custom message type will have mixed results, ranging from panics, to unspecified behavior, to working correctly. The proper way for custom message types to perfectly work with this module is for the custom type to implement the protoreflect.ProtoMessage interface, which allows a given message implementation to self-describe how its contents should be introspected.

Since some widely depended upon modules have custom message types not generated by protoc-gen-go, we expand support in this module's runtime to be able to handle those aberrant message types. This change should reduce the probability that using one of those modules with this module results in a panic and instead have reasonably correct behavior.

This support is not covered by this project's compatibility guarantee and may be modified or removed in a future version of this module without notice. Custom types must move to supporting protobuf reflection to ensure long-term compatibility.

Upcoming breakage changes

This release makes some potentially disruptive changes that have been announced since the v1.20.0 release on March 2nd, 2020. Not all breaking changes mentioned in the v1.20.0 have been made and may s...

v1.25.0

• Overview
• Notable changes
  • Specialized support for well-known types
• Upcoming breakage changes

Overview

This release provides specialized support for well-known types.

Notable changes

New features:

• CL/225298: types/known: implement first-class WKT support
  • Related follow-up changes: CL/237920, CL/237997, CL/238007, CL/238009
• CL/237219: types/dynamicpb: add NewEnumType

Minor changes:

• CL/238002: reflect/protoreflect: optimize Name.IsValid and FullName.IsValid
• CL/239339: cmd/protoc-gen-go: use of --version flag exits with 0

Bug fixes:

• CL/236998: encoding/protojson: strict validation of FieldMask serialization

Specialized support for well-known types

This release provides specialized support for well-known types by directly generating methods and functions into the generated package for certain well-known types that improve the usability of such types. The additionally generated APIs are listed below.

• Package anypb: New, MarshalFrom, UnmarshalTo, UnmarshalNew, Any.MessageIs, Any.MessageName, Any.MarshalFrom, Any.UnmarshalTo, Any.UnmarshalNew

• Package timestamppb: Now, New, Timestamp.AsTime, Timestamp.IsValid, Timestamp.CheckValid

• Package durationpb: New, Duration.AsDuration, Duration.IsValid, Duration.CheckValid

• Package structpb: NewStruct, Struct.AsMap, Struct.MarshalJSON, Struct.UnmarshalJSON, NewList, ListValue.AsSlice, ListValue.MarshalJSON, ListValue.UnmarshalJSON, NewValue, NewNullValue, NewBoolValue, NewNumberValue, NewStringValue, NewStructValue, NewListValue, Value.AsInterface, Value.MarshalJSON, Value.UnmarshalJSON

• Package fieldmaskpb New,, Union,, Intersect,, FieldMask.IsValid,, FieldMask.Append,, FieldMask.Normalize,

• Package wrapperspb Bool, Int32, Int64, UInt32, UInt64, Float, Double, String, Bytes

Upcoming breakage changes

There are no new upcoming breaking changes to announce in this release.

As a reminder, the following have already been announced and may take effect in the near future:

• Deprecations announced on 2020-03-02 with the v1.20.0 release

v1.24.0

• Overview
• Notable changes
  • Centralized host for all well-known types
• Upcoming breakage changes

Overview

This release moves the generation of well-known types into this module.

Notable changes

New features:

• CL/234937: all: move well-known types

Minor changes:

• CL/235297: all: add weak dependency on google.golang.org/genproto module

Bug fixes:

• CL/234697: testing/protopack: fix format precision

Centralized host for all well-known types

This release moves the generation of well-known types (e.g., FieldMask) into this module. Previously, some of the well-known types were generated into the google.golang.org/genproto module. This sets the stage for a future release of this module to provide first-class support for the well-known types.

As a consequence of this migration, this module incurs a weak dependency on the google.golang.org/genproto module. This dependency may be dropped in the distant future, but it will persist for some time to ensure that users of this module use a sufficiently new version of google.golang.org/genproto (if they happen to depend on it) so as to avoid duplicate registration problems with regard to the well-known types.

Upcoming breakage changes

There are no new upcoming breaking changes to announce in this release.

As a reminder, the following have already been announced and may take effect in the near future:

• Deprecations announced on 2020-03-02 with the v1.20.0 release

v1.23.0

• Overview
• Notable changes
• Upcoming breakage changes

Overview

This release adds some minor new API to encoding/prototext and compiler/protogen.

Notable changes

New features:

• CL/233077: encoding/prototext: adjust handling of invalid UTF-8
• CL/232239: compiler/protogen: add (*GeneratedFile).Unskip

Minor changes:

• CL/232339: all: improve panic messages for better debugability
• CL/232937: internal/impl: avoid inlining fixed coderFieldInfo array

Bug fixes:

• CL/232338: compiler/protogen: avoid suggesting faulty go_package option

Upcoming breakage changes

There are no new upcoming breaking changes to announce in this release.

As a reminder, the following have already been announced and may take effect in the near future:

• Deprecations announced on 2020-03-02 with the v1.20.0 release

v1.22.0

• Overview
• Notable changes
  • Proto3 optional semantics
  • Adjustments to ease migration
• Upcoming breakage changes

Overview

This release adds support for the experimental proto3 optional semantics coming in the v3.12.0 release of the protobuf language, adjusts the API to ease migration from the old module to the new module, and fixes some minor bugs.

Notable changes

New features:

• CL/230698: all: implement support for proto3 optional semantics
• CL/229157: proto: add RangeExtensions and adjust HasExtension and GetExtension

Minor changes:

• CL/228837: all: consistently treat nil message interface as an empty read-only message
• CL/228838: proto: never return nil []byte from Marshal when successful
• CL/229558: all: improve extension validation
• CL/229277: internal/impl: inline coderInfoFields for better cache locality

Bug fixes:

• CL/229278: internal/impl: fix off-by-one error in message initialization

Proto3 optional semantics

Protocol buffers in v3.12.0 is gaining experimental support to represent true presence for scalars in the proto3 syntax. By default, scalars in proto3 lack presence, so users are unable to distinguish between an unpopulated field and whether the zero value was explicitly stored into that field. With this new language feature, users can explicitly specify the optional keyword for scalars to indicate that presence information should be preserved for that field.

This release adds support to the generator and the runtime for this feature. Generation of proto3 optional fields are represented as pointers to scalar types (similar to proto2 optional scalars).

Adjustments to ease migration

A few minor changes were made to the API to unify the behavior between the old github.com/golang/protobuf module and the new google.golang.org/protobuf module. This should aid users migrating from the former to the latter.

Changes:

• Adjusted all functions to treat a nil proto.Message as an untyped, read-only, empty message and provide sensible results given that semantic.
• Adjusted proto.HasExtension to always report false if the extension descriptor does not extend the parent message (as opposed to panicking previously).
• Added proto.RangeExtensions as a replacement for the legacy proto.ClearAllExtensions and proto.ExtensionDescs functions.
• Fixed protoreflect.ExtensionType.IsValidInterface to never panic and properly report whether the given value is of the right type.

Upcoming breakage changes

There are no new upcoming breaking changes to announce in this release.

As a reminder, the following have already been announced and may take effect in the near future:

• Deprecations announced on 2020-03-02 with the v1.20.0 release

v1.21.0

• Overview
• Notable changes
  • Code generator flags
  • Package protowire
  • Package protopack
  • Package protocmp
• Upcoming breakage changes

Overview

This release contains some improvements to the usability of the protoc-gen-go tool and the release of several packages that make it easier to interact directly with the binary wire format.

Notable changes

New features:

• CL/219298: compiler/protogen: add module= generator option
• CL/219597: compiler/protogen: make paths=import work with M overrides
• CL/223819: compiler/protogen: allow specifying the package name with M flags
• CL/219838: encoding/protowire: make package publicly available
• CL/219837: testing/protopack: make package publicly available
• CL/221432: testing/protocmp: add SortRepeated and SortRepeatedFields
• CL/226237: reflect/protodesc: add NewFiles

Minor changes:

• CL/223817: types/dynamic: make Message implement legacy message interface
• CL/223857: types/dynamicpb: fix message Zero return a read-only type

Code generator flags

A new module flag is added to the generator that controls stripping of a prefix from the generated filenames. This option enables protoc-gen-go to better generate .pb.go files for a workflow that depends on Go modules rather than a centralized Go path.

For example, suppose a protos/source.proto file has a go_package option specified as follows:

option go_proto = "github.com/example/project/foo";

and that the module root for this project is github.com/example/project.

If invoked from the module root, this flag could be potentially used as follows:

protoc --go_out=module=github.com/example/project:. protos/source.proto

This would emit a file located foo/source.pb.go rather than being located at github.com/example/project/foo/source.pb.go.

Furthermore, two minor changes have been made to the M flag, which can be used to specify the Go package path corresponding to a given .proto file:

• When used with the paths=import option, the implementation now correctly respects the effect of that option and outputs the file as import-relative, rather than always being source-relative.
• If the Go package path is provided with a ; delimiter, it can be used to specify the exact package name for the generated package. It's syntax is identical to that of the go_package option.

Package protowire

The protowire package provides functionality for parsing and packing the binary wire format. Effective use of this package requires that the user has an understanding of the wire format itself. The package is written in a performance sensitive way and currently serves as the underlying implementation of the protobuf runtime. It is the replacement for methods previously provided on the Buffer type in the legacy proto package.

Example of how to parse wire data for an encoded message:

import "google.golang.org/protobuf/encoding/protowire"

var b []byte = ... // encoded bytes of a protobuf message
for len(b) > 0 {
    num, typ, n := protowire.ConsumeField(b)
    if n < 0 {
        log.Fatalf("parse error: %v", protowire.ParseError(n))
    }
    log.Printf("FieldNumber:%d  WireType:%d  WireData:%x", num, typ, b[:n])
    b = b[n:]
}

Using the encoded output of the following protopack example, this produces:

FieldNumber:1  WireType:2  WireData:0a0d48656c6c6f2c20776f726c6421
FieldNumber:2  WireType:0  WireData:10f6ffffffffffffffff01
FieldNumber:3  WireType:2  WireData:1a0ccdcc8c3fcdcc0c4033335340

Package protopack

The protopack package is intended to be used for debugging the binary wire format or assist in the construction of testdata in the wire format. Effective use of this package requires that the user has an understanding of the wire format itself. The protopack package essentially provides a set of types that represents the wire data as a concrete syntax tree.

Given this proto3 message definition:

message MyMessage {
    string field1 = 1;
    int64 field2 = 2;
    repeated float32 field3 = 3;
}

Valid wire data for the message can be constructed as such:

import . "google.golang.org/protobuf/testing/protopack"

b := Message{
    Tag{1, BytesType}, String("Hello, world!"),
    Tag{2, VarintType}, Varint(-10),
    Tag{3, BytesType}, LengthPrefix{
        Float32(1.1), Float32(2.2), Float32(3.3),
    },
}.Marshal()

This results in the following output data:

0x0000  0a 0d 48 65 6c 6c 6f 2c  20 77 6f 72 6c 64 21 10  |..Hello, world!.|
0x0010  f6 ff ff ff ff ff ff ff  ff 01 1a 0c cd cc 8c 3f  |...............?|
0x0020  cd cc 0c 40 33 33 53 40                           |...@33S@|

Package protocmp

The SortRepeated and SortRepeatedFields options are added to the protocmp package. These options enable repeated fields to be sorted so that they are functionally treated as multisets.

The SortRepeated option sorts repeated fields of a specific element type. This option requires the user to provide a less function where the field type is inferred from the provided function type.

The SortRepeatedFields option sorts a set of repeated fields on a specific message type. This option requires the user to specify the set of fields by name. An arbitrary sort ordering is chosen for each specified repeated field.

Upcoming breakage changes

There are no new upcoming breaking changes to announce in this release.

As a reminder, the following breaking changes have already been announced and may take effect 6 months after they have first been announced:

• Announced on 2020-03-02 with the v1.20.0 release:
  • Enforcement of namespace uniqueness
  • Requirement of the go_package option
  • Deprecation of certain flags for protoc-gen-go
  • Deprecation of certain APIs in generated code

v1.20.1

Notable changes since v1.20.0:

• CL/204804: bug fix for dead-lock in the global type registry
• CL/221803: minor performance improvements for Marshal and Size operations
• CL/222678: fix signature of pseudo-internal MarshalState and UnmarshalState methods

v1.20.0

• Overview
  • Backwards compatibility
  • Package index
• Notable changes
  • Message type
  • Reflection
  • Wire serialization
  • JSON serialization
  • Text serialization
  • Extensions
  • Nil values
  • Errors
  • Generated code
  • gRPC support
  • Performance
• Upcoming breakage changes
  • Enforcement of namespace uniqueness
  • Required go_package option
  • Deprecation of protoc-gen-go flags
  • Deprecation of generated APIs

Overview

The google.golang.org/protobuf module is a major revision of the Go bindings for protocol buffers. The API has been completely overhauled with a focus on simplicity and clarity. Deprecated types and functions are removed, complicated packages are split into more manageable pieces, and internal details are separated from the public API. Particular attention was made to ensure that this implementation complies with the protocol buffer language as closely as possible to ensure proper interoperability between the Go implementation and implementations for other languages.

The first version of this module is v1.20.0. The version number is deliberately chosen to avoid overlap with existing and anticipated versions of the github.com/golang/protobuf module. This enables the version alone to unambiguously distinguish between the two modules.

The minimally supported version of Go is v1.9.

Backwards compatibility

The API in this module is not drop-in compatible with the earlier github.com/golang/protobuf module. However, the new API does strive when sensible to be compatible for easier migration.

Version v1.4.0 of the github.com/golang/protobuf module implements most of the existing API in terms of this module, which provides a number of benefits:

• Shared type registry: It ensures that the two modules share a unified type registry, which is necessary for dynamic usages of protocol buffers to work properly.
• Automatic improvements: It provides the older module with a degree of automatic upgrading whenever bug fixes and improvements are made to the newer module.
• Long-term maintainability: It keeps the older module more maintainable since it is largely a thin shim over the newer module.

Some existing features present in the github.com/golang/protobuf module are currently unavailable in this module:

• Manipulation of the wire format: The older module provides functionality for directly manipulating the raw wire format through methods on the proto.Buffer type.
• Helper functions for well-known types: The older module provides the ptypes package containing helper functions for interacting with the well-known types such as Any or Timestamp messages.

We plan to add these features to this module in the near future and improve upon them.

Package index

Summary of the packages provided by this module:

• proto: Package proto provides functions operating on protobuf messages such as cloning, merging, and checking equality, as well as binary serialization.
• encoding/protojson: Package protojson serializes protobuf messages as JSON.
• encoding/prototext: Package prototext serializes protobuf messages as the text format.
• reflect/protoreflect: Package protoreflect provides interfaces to dynamically manipulate protobuf messages.
• reflect/protoregistry: Package protoregistry provides data structures to register and lookup protobuf descriptor types.
• reflect/protodesc: Package protodesc provides functionality for converting descriptorpb.FileDescriptorProto messages to/from the reflective protoreflect.FileDescriptor.
• testing/protocmp: Package protocmp provides protobuf specific options for the cmp package.
• testing/prototest: Package prototest exercises the protobuf reflection implementation for concrete message types.
• types/dynamicpb: Package dynamicpb creates protobuf messages at runtime from protobuf descriptors.
• types/known/anypb: Package anypb is the generated package for google/protobuf/any.proto.
• types/known/emptypb: Package emptypb is the generated package for google/protobuf/empty.proto.
• types/known/timestamppb: Package timestamppb is the generated package for google/protobuf/timestamp.proto.
• types/known/durationpb: Package durationpb is the generated package for google/protobuf/duration.proto.
• types/known/wrapperspb: Package wrapperspb is the generated package for google/protobuf/wrappers.proto.
• types/known/structpb: Package structpb is the generated package for google/protobuf/struct.proto.
• types/descriptorpb: Package descriptorpb is the generated package for google/protobuf/descriptor.proto.
• types/pluginpb: Package pluginpb is the generated package for google/protobuf/compiler/plugin.proto.
• compiler/protogen: Package protogen provides support for writing protoc plugins.
• cmd/protoc-gen-go: The protoc-gen-go binary is a protoc plugin to generate a Go protocol buffer package.

Notable changes

This section summarizes notable changes in this release relative to v1.3.4 of the github.com/golang/protobuf module.

Message type

The definition of the proto.Message interface type has changed (see v1 Message and v2 Message). The new type definition contains a behaviorally-complete specification of a protobuf message. All functions in this module that operate on a proto.Message can be used with values of any concrete type which properly implements this interface.

The github.com/golang/protobuf/proto package contains functions to convert between the old and new proto.Message types (see MessageV1 and MessageV2). These functions are guaranteed to work with types generated by protoc-gen-go, however there is no guarantee that it handles hand-crafted messages or messages generated by different generators.

Reflection

A key feature of the new Go protobuf API is support for protobuf reflection, which is the ability to programmatically interact with a message at runtime. The ProtoReflect method implemented by every proto.Message returns a reflective view of that message. Using protobuf reflection instead of Go reflection is generally easier since it closely follows the protobuf data model, and also more maintainable since it relies on a stable API that is agnostic to the exact underlying implementation of a message. On the other hand, using Go reflection on protobuf messages often needs to make low-level assumptions about a particular implementation of protobuf messages that may not be stable (e.g., assuming that the Go struct field ordering exactly matches the proto message field ordering).

See the protoreflect package for more details.

Wire serialization

The API for wire serialization has changed slightly for better usability. Optional arguments for controlling serialization are specified through the proto.MarshalOptions and proto.UnmarshalOptions types. Both...