Protocol Buffers v3.7.0-rc.3

Merge pull request #5766 from TeBoring/3.7.x-fix

Update version number to 3.7.0rc3

Protocol Buffers v3.7.0rc2

Fixed infinite recursion problem with Java GeneratedMessageV3 (#5657)

GeneratedMessageV3#setUnknownFieldsProto3 was trying to delegate to
setUnknownFields but was inadvertently resulting in infinite recursion.
This commit makes setUnknownFields and setUnknownFieldsProto3 delegate
to a common private method to fix the problem and avoid confusion.

Protocol Buffers v3.7.0rc1

Merge pull request #5644 from TeBoring/fix-4765

Convert integer to string if field is string field in json

Protocol Buffers v3.6.1

C++

• Introduced workaround for Windows issue with std::atomic and std::once_flag initialization (#4777, #4773)

PHP

• Added compatibility with PHP 7.3 (#4898)

Ruby

• Fixed Ruby crash involving Any encoding (#4718)

Protocol Buffers v3.6.0

General

• We are moving protobuf repository to its own github organization (see #4796). Please let us know what you think about the move by taking this survey: https://docs.google.com/forms/d/e/1FAIpQLSeH1ckwm6ZrSfmtrOjRwmF3yCSWQbbO5pTPqPb6_rUppgvBqA/viewform

C++

• Starting from this release, we now require C++11. For those we cannot yet upgrade to C++11, we will try to keep the 3.5.x branch updated with critical bug fixes only. If you have any concerns about this, please comment on issue #2780.
• Moved to C++11 types like std::atomic and std::unique_ptr and away from our old custom-built equivalents.
• Added support for repeated message fields in lite protos using implicit weak fields. This is an experimental feature that allows the linker to strip out more unused messages than previously was possible.
• Fixed SourceCodeInfo for interpreted options and extension range options.
• Fixed always_print_enums_as_ints option for JSON serialization.
• Added support for ignoring unknown enum values when parsing JSON.
• Create std::string in Arena memory.
• Fixed ValidateDateTime to correctly check the day.
• Fixed bug in ZeroCopyStreamByteSink.
• Various other cleanups and fixes.

Java

• Dropped support for Java 6.
• Added a UTF-8 decoder that uses Unsafe to directly decode a byte buffer.
• Added deprecation annotations to generated code for deprecated oneof fields.
• Fixed map field serialization in DynamicMessage.
• Cleanup and documentation for Java Lite runtime.
• Various other fixes and cleanups
• Fixed unboxed arraylists to handle an edge case
• Improved performance for copying between unboxed arraylists
• Fixed lite protobuf to avoid Java compiler warnings
• Improved test coverage for lite runtime
• Performance improvements for lite runtime

Python

• Fixed bytes/string map key incompatibility between C++ and pure-Python implementations (issue #4029)
• Added __init__.py files to compiler and util subpackages
• Use /MT for all Windows versions
• Fixed an issue affecting the Python-C++ implementation when used with Cython (issue #2896)
• Various text format fixes
• Various fixes to resolve behavior differences between the pure-Python and Python-C++ implementations

PHP

• Added php_metadata_namespace to control the file path of generated metadata file.
• Changed generated classes of nested message/enum. E.g., Foo.Bar, which previously generates Foo_Bar, now generates Foo/Bar
• Added array constructor. When creating a message, users can pass a php array whose content is field name to value pairs into constructor. The created message will be initialized according to the array. Note that message field should use a message value instead of a sub-array.
• Various bug fixes.

Objective-C

• We removed some helper class methods from GPBDictionary to shrink the size of the library, the functionary is still there, but you may need to do some specific +alloc / -init… methods instead.
• Minor improvements in the performance of object field getters/setters by avoiding some memory management overhead.
• Fix a memory leak during the raising of some errors.
• Make header importing completely order independent.
• Small code improvements for things the undefined behaviors compiler option was flagging.

Ruby

• Added ruby_package file option to control the module of generated class.
• Various bug fixes.

Javascript

• Allow setting string to int64 field.

Csharp

• Unknown fields are now parsed and then sent back on the wire. They can be discarded at parse time via a CodedInputStream option.
• Movement towards working with .NET 3.5 and Unity
• Expression trees are no longer used
• AOT generics issues in Unity/il2cpp have a workaround (see commit 1b219a1 for details)
• Floating point values are now compared bitwise (affects NaN value comparisons)
• The default size limit when parsing is now 2GB rather than 64MB
• MessageParser now supports parsing from a slice of a byte array
• JSON list parsing now accepts null values where the underlying proto representation does

Protocol Buffers v3.5.1

Planned Future Changes

• Make C++ implementation C++11 only: we plan to require C++11 to build protobuf code starting from 3.6.0 release, after unknown fields semantic changes are finished. Please join this github issue to provide your feedback.

protoc

• Fixed a bug introduced in 3.5.0 and protoc in Windows now accepts non-ascii characters in paths again.

C++

• Removed several usages of C++11 features in the code base.
• Fixed some compiler warnings.

PHP

• Fixed memory leak in C-extension implementation.
• Added discardUnknokwnFields API.
• Removed duplicatd typedef in C-extension headers.
• Avoided calling private php methods (timelib_update_ts).
• Fixed Any.php to use fully-qualified name for DescriptorPool.

Ruby

• Added Google_Protobuf_discard_unknown for discarding unknown fields in
  messages.

C#

• Unknown fields are now preserved by default.
• Floating point values are now bitwise compared, affecting message equality check and Contains() API in map and repeated fields.

Protocol Buffers v3.5.0

Planned Future Changes

• Make C++ implementation C++11 only: we plan to require C++11 to build protobuf code starting from 3.5.0 or 3.6.0 release, after unknown fields semantic changes are finished. Please join this github issue to provide your feedback.

General

• Unknown fields are now preserved in proto3 for most of the language implementations for proto3 by default. See the per-language section for details.
• reserve keyword are now supported in enums

C++

• Proto3 messages are now preserving unknown fields by default. If you rely on unknowns fields being dropped. Please use DiscardUnknownFields() explicitly.
• Deprecated the unsafe_arena_release_* and unsafe_arena_add_allocated_* methods for string fields.
• Added move constructor and move assignment to RepeatedField, RepeatedPtrField and google::protobuf::Any.
• Added perfect forwarding in Arena::CreateMessage
• In-progress experimental support for implicit weak fields with lite protos. This feature allows the linker to strip out more unused messages and reduce binary size.
• Various performance optimizations.

Java

• Proto3 messages are now preserving unknown fields by default. If you’d like to drop unknown fields, please use the DiscardUnknownFieldsParser API. For example:

      Parser<Foo> parser = DiscardUnknownFieldsParser.wrap(Foo.parser());
      Foo foo = parser.parseFrom(input);

• Added a new CodedInputStream decoder for Iterable<ByteBuffer> with direct ByteBuffers.
• TextFormat now prints unknown length-delimited fields as messages if possible.
• FieldMaskUtil.merge() no longer creates unnecessary empty messages when a message field is unset in both source message and destination message.
• Various performance optimizations.

Python

• Proto3 messages are now preserving unknown fields by default. Use message.DiscardUnknownFields() to drop unknown fields.
• Add FieldDescriptor.file in generated code.
• Add descriptor pool FindOneofByName in pure python.
• Change unknown enum values into unknown field set .
• Add more Python dict/list compatibility for Struct/ListValue.
• Add utf-8 support for text_format.Merge()/Parse().
• Support numeric unknown enum values for proto3 JSON format.
• Add warning for Unexpected end-group tag in cpp extension.

PHP

• Proto3 messages are now preserving unknown fields.
• Provide well known type messages in runtime.
• Add prefix ‘PB’ to generated class of reserved names.
• Fixed all conformance tests for encode/decode json in php runtime. C extension needs more work.

Objective-C

• Fixed some issues around copying of messages with unknown fields and then mutating the unknown fields in the copy.

C#

• Added unknown field support in JsonParser.
• Fixed oneof message field merge.
• Simplify parsing messages from array slices.

Ruby

• Unknown fields are now preserved by default.
• Fixed several bugs for segment fault.

Javascript

• Decoder can handle both paced and unpacked data no matter how the proto is defined.
• Decoder now accept long varint for 32 bit integers.

Protocol Buffers v3.4.1

This is mostly a bug fix release on runtime packages. It is safe to use 3.4.0 protoc packages for this release.

• Fixed the missing files in 3.4.0 tarballs, affecting windows and cmake users.
• C#: Fixed dotnet target platform to be net45 again.
• Ruby: Fixed a segmentation error when using maps in multi-threaded cases.
• PHP: php_generic_service file level option tag number (in descriptor.proto) has been reassigned to avoid conflicts.

Protocol Buffers v3.4.0

Planned Future Changes

• Preserve unknown fields in proto3: We are going to bring unknown fields back into proto3. In this release, some languages start to support preserving unknown fields in proto3, controlled by flags/options. Some languages also introduce explicit APIs to drop unknown fields for migration. Please read the change log sections by languages for details. See general timeline and plan and issues and discussions

• Make C++ implementation C++11 only: we plan to require C++11 to build protobuf code starting from 3.5.0 or 3.6.0 release, after unknown fields semantic changes are finished. Please join this github issue to provide your feedback.

General

• Extension ranges now accept options and are customizable.
• reserve keyword now supports max in field number ranges, e.g. reserve 1000 to max;

C++

• Proto3 messages are now able to preserve unknown fields. The default behavior is still to drop unknowns, which will be flipped in a future release. If you rely on unknowns fields being dropped. Please use Message::DiscardUnknownFields() explicitly.

• Packable proto3 fields are now packed by default in serialization.

• Following C++11 features are introduced when C++11 is available:

  • move-constructor and move-assignment are introduced to messages
  • Repeated fields constructor now takes std::initializer_list
  • rvalue setters are introduced for string fields

• Experimental Table-Driven parsing and serialization available to test. To enable it, pass in table_driven_parsing table_driven_serialization protoc generator flags for C++

  $ protoc --cpp_out=table_driven_parsing,table_driven_serialization:./ test.proto

• lite generator parameter supported by the generator. Once set, all generated files, use lite runtime regardless of the optimizer_for setting in the .proto file.

• Various optimizations to make C++ code more performant on PowerPC platform

• Fixed maps data corruption when the maps are modified by both reflection API and generated API.

• Deterministic serialization on maps reflection now uses stable sort.

• file() accessors are introduced to various *Descriptor classes to make writing template function easier.

• ByteSize() and SpaceUsed() are deprecated.Use ByteSizeLong() and SpaceUsedLong() instead

• Consistent hash function is used for maps in DEBUG and NDEBUG build.

• "using namespace std" is removed from stubs/common.h

• Various performance optimizations and bug fixes

Java

• Introduced new parser API DiscardUnknownFieldsParser in preparation of proto3 unknown fields preservation change. Users who want to drop unknown fields should migrate to use this new parser API.
  For example:

  Parser<Foo> parser = DiscardUnknownFieldsParser.wrap(Foo.parser());
  Foo foo = parser.parseFrom(input);

• Introduced new TextFormat API printUnicodeFieldValue() that prints field value without escaping unicode characters.

• Added Durations.compare(Duration, Duration) and Timestamps.compare(Timestamp, Timestamp).

• JsonFormat now accepts base64url encoded bytes fields.

• Optimized CodedInputStream to do less copies when parsing large bytes fields.

• Optimized TextFormat to allocate less memory when printing.

Python

• SerializeToString API is changed to SerializeToString(self, **kwargs), deterministic parameter is accepted for deterministic serialization.
• Added sort_keys parameter in json format to make the output deterministic.
• Added indent parameter in json format.
• Added extension support in json format.
• Added __repr__ support for repeated field in cpp implementation.
• Added file in FieldDescriptor.
• Added pretty-print filter to text format.
• Services and method descriptors are always printed even if generic_service option is turned off.
• Note: AppEngine 2.5 is deprecated on June 2017 that AppEngine 2.5 will never update protobuf runtime. Users who depend on AppEngine 2.5 should use old protoc.

PHP

• Support PHP generic services. Specify file option php_generic_service=true to enable generating service interface.
• Message, repeated and map fields setters take value instead of reference.
• Added map iterator in c extension.
• Support json  encode/decode.
• Added more type info in getter/setter phpdoc
• Fixed the problem that c extension and php implementation cannot be used together.
• Added file option php_namespace to use custom php namespace instead of package.
• Added fluent setter.
• Added descriptor API in runtime for custom encode/decode.
• Various bug fixes.

Objective-C

• Fix for GPBExtensionRegistry copying and add tests.
• Optimize GPBDictionary.m codegen to reduce size of overall library by 46K per architecture.
• Fix some cases of reading of 64bit map values.
• Properly error on a tag with field number zero.
• Preserve unknown fields in proto3 syntax files.
• Document the exceptions on some of the writing apis.

C#

• Implemented IReadOnlyDictionary<K,V> in MapField<K,V>
• Added TryUnpack method for Any message in addition to Unpack.
• Converted C# projects to MSBuild (csproj) format.

Ruby

• Several bug fixes.

Javascript

• Added support of field option js_type. Now one can specify the JS type of a 64-bit integer field to be string in the generated code by adding option [jstype = JS_STRING] on the field.

Protocol Buffers v3.3.0

Planned Future Changes

• There are some changes that are not included in this release but are planned for the near future:
  • Preserve unknown fields in proto3: please read this doc for the timeline and follow up this github issue for discussion.
  • Make C++ implementation C++11 only: we plan to require C++11 to build protobuf code starting from 3.4.0 or 3.5.0 release. Please join this github issue to provide your feedback.

C++

• Fixed map fields serialization of DynamicMessage to correctly serialize both key and value regardless of their presence.
• Parser now rejects field number 0 correctly.
• New API Message::SpaceUsedLong() that’s equivalent to Message::SpaceUsed() but returns the value in size_t.
• JSON support
  • New flag always_print_enums_as_ints in JsonPrintOptions.
  • New flag preserve_proto_field_names in JsonPrintOptions. It will instruct the JSON printer to use the original field name declared in the .proto file instead of converting them to lowerCamelCase when printing JSON.
  • JsonPrintOptions.always_print_primtive_fields now works for oneof message fields.
  • Fixed a bug that doesn’t allow different fields to set the same json_name value.
  • Fixed a performance bug that causes excessive memory copy when printing large messages.
• Various performance optimizations.

Java

• Map field setters eagerly validate inputs and throw NullPointerExceptions as appropriate.
• Added ByteBuffer overloads to the generated parsing methods and the Parser interface.
• proto3 enum's getNumber() method now throws on UNRECOGNIZED values.
• Output of JsonFormat is now locale independent.

Python

• Added FindServiceByName() in the pure-Python DescriptorPool. This works only for descriptors added with DescriptorPool.Add(). Generated descriptor_pool does not support this yet.
• Added a descriptor_pool parameter for parsing Any in text_format.Parse().
• descriptor_pool.FindFileContainingSymbol() now is able to find nested extensions.
• Extending empty [] to repeated field now sets parent message presence.

PHP

• Added file option php_class_prefix. The prefix will be prepended to all generated classes defined in the file.
• When encoding, negative int32 values are sign-extended to int64.
• Repeated/Map field setter accepts a regular PHP array. Type checking is done on the array elements.
• encode/decode are renamed to serializeToString/mergeFromString.
• Added mergeFrom, clear method on Message.
• Fixed a bug that oneof accessor didn’t return the field name that is actually set.
• C extension now works with php7.
• This is the first GA release of PHP. We guarantee that old generated code can always work with new runtime and new generated code.

Objective-C

• Fixed help for GPBTimestamp for dates before the epoch that contain fractional seconds.
• Added GPBMessageDropUnknownFieldsRecursively() to remove unknowns from a message and any sub messages.
• Addressed a threading race in extension registration/lookup.
• Increased the max message parsing depth to 100 to match the other languages.
• Removed some use of dispatch_once in favor of atomic compare/set since it needs to be heap based.
• Fixes for new Xcode 8.3 warnings.

C#

• Fixed MapField.Values.CopyTo, which would throw an exception unnecessarily if provided exactly the right size of array to copy to.
• Fixed enum JSON formatting when multiple names mapped to the same numeric value.
• Added JSON formatting option to format enums as integers.
• Modified RepeatedField to implement IReadOnlyList.
• Introduced the start of custom option handling; it's not as pleasant as it might be, but the information is at least present. We expect to extend code generation to improve this in the future.
• Introduced ByteString.FromStream and ByteString.FromStreamAsync to efficiently create a ByteString from a stream.
• Added whole-message deprecation, which decorates the class with [Obsolete].

Ruby

• Fixed Message#to_h for messages with map fields.
• Fixed memcpy() in binary gems to work for old glibc, without breaking the build for non-glibc libc’s like musl.

Javascript

• Added compatibility tests for version 3.0.0.
• Added conformance tests.
• Fixed serialization of extensions: we need to emit a value even if it is falsy (like the number 0).
• Use closurebuilder.py in favor of calcdeps.py for compiling JavaScript.