core: duplicate symbols coming from protobuf

[rebello95]

Overview

With the latest upstream Envoy update, we're seeing quite a few duplicate symbols coming from the final Envoy.framework. These symbols seem to be coming from protobuf. Here are a few examples:

05:06:52.047624 Found symbol '__ZN6google8protobuf28UninterpretedOption_NamePartC1ERKS1_' in multiple binaries: Envoy, Envoy'
05:06:52.047630 Found symbol '__ZNK6google8protobuf19EnumDescriptorProto13IsInitializedEv' in multiple binaries: Envoy, Envoy'
05:06:52.047839 Found symbol '__ZN6google8protobuf12OneofOptions9MergeFromERKS1_' in multiple binaries: Envoy, Envoy'
05:06:52.047865 Found symbol '_descriptor_table_google_2fprotobuf_2fdescriptor_2eproto' in multiple binaries: Envoy, Envoy'
05:06:52.047873 Found symbol '__ZN6google8protobuf11UInt64Value9MergeFromERKS1_' in multiple binaries: Envoy, Envoy'
05:06:52.047880 Found symbol '__ZNK6google8protobuf19UninterpretedOption39InternalSerializeWithCachedSizesToArrayEPhPNS0_2io19EpsCopyOutputStreamE' in multiple binaries: Envoy, Envoy'
05:06:52.047886 Found symbol '__ZN6google8protobuf19FileDescriptorProto8CopyFromERKS1_' in multiple binaries: Envoy, Envoy'
05:06:52.047895 Found symbol '__ZN6google8protobuf29DescriptorProto_ReservedRangeC2EPNS0_5ArenaE' in multiple binaries: Envoy, Envoy'
05:06:52.047906 Found symbol '__ZNK6google8protobuf28GeneratedCodeInfo_Annotation12ByteSizeLongEv' in multiple binaries: Envoy, Envoy'
05:06:52.047972 Found symbol '__ZNK6google8protobuf20FieldDescriptorProto12ByteSizeLongEv' in multiple binaries: Envoy, Envoy'
05:06:52.048006 Found symbol '__ZN6google8protobuf12FieldOptions10JSType_MINE' in multiple binaries: Envoy, Envoy'
05:06:52.048015 Found symbol '__ZN6google8protobuf28UninterpretedOption_NamePartC1EPNS0_5ArenaE' in multiple binaries: Envoy, Envoy'
05:06:52.048060 Found symbol '__ZTSN6google8protobuf11UInt64ValueE' in multiple binaries: Envoy, Envoy'
05:06:52.048072 Found symbol '__ZN6google8protobuf20FieldDescriptorProto12TYPE_FIXED64E' in multiple binaries: Envoy, Envoy'
05:06:52.048164 Found symbol '_scc_info_Empty_google_2fprotobuf_2fempty_2eproto' in multiple binaries: Envoy, Envoy'
05:06:52.048184 Found symbol '_scc_info_OneofOptions_google_2fprotobuf_2fdescriptor_2eproto' in multiple binaries: Envoy, Envoy'
05:06:52.048201 Found symbol '__ZNK6google8protobuf16EnumValueOptions39InternalSerializeWithCachedSizesToArrayEPhPNS0_2io19EpsCopyOutputStreamE' in multiple binaries: Envoy, Envoy'
05:06:52.048218 Found symbol '__ZN6google8protobuf11UInt32Value5ClearEv' in multiple binaries: Envoy, Envoy'
05:06:52.048261 Found symbol '__ZNK6google8protobuf11DoubleValue13IsInitializedEv' in multiple binaries: Envoy, Envoy'
05:06:52.048406 Found symbol '__ZN6google8protobuf20OneofDescriptorProtoD2Ev' in multiple binaries: Envoy, Envoy'
05:06:52.048434 Found symbol '__ZN6google8protobuf29DescriptorProto_ReservedRange21InitAsDefaultInstanceEv' in multiple binaries: Envoy, Envoy'
05:06:52.048448 Found symbol '__ZN6google8protobuf28GeneratedCodeInfo_AnnotationD2Ev' in multiple binaries: Envoy, Envoy'
05:06:52.048455 Found symbol '__ZN6google8protobuf23SourceCodeInfo_Location5ClearEv' in multiple binaries: Envoy, Envoy'
05:06:52.048461 Found symbol '__ZN6google8protobuf5Arena18CreateMaybeMessageINS0_30DescriptorProto_ExtensionRangeEJEEEPT_PS1_DpOT0_' in multiple binaries: Envoy, Envoy'
05:06:52.048637 Found symbol '__ZNK6google8protobuf13MethodOptions12ByteSizeLongEv' in multiple binaries: Envoy, Envoy'
05:06:52.048657 Found symbol '__ZN6google8protobuf17GeneratedCodeInfoD2Ev' in multiple binaries: Envoy, Envoy'
05:06:52.048739 Found symbol '__ZN6google8protobuf28_BoolValue_default_instance_E' in multiple binaries: Envoy, Envoy'
05:06:52.048756 Found symbol '__ZN6google8protobuf8Duration8CopyFromERKNS0_7MessageE' in multiple binaries: Envoy, Envoy'
05:06:52.048763 Found symbol '__ZN6google8protobuf11EnumOptionsD2Ev' in multiple binaries: Envoy, Envoy'
05:06:52.048770 Found symbol '__ZN6google8protobuf13MethodOptions26IdempotencyLevel_ARRAYSIZEE' in multiple binaries: Envoy, Envoy'
05:06:52.048911 Found symbol '__ZNK6google8protobuf19FileDescriptorProto13SetCachedSizeEi' in multiple binaries: Envoy, Envoy'
05:06:52.048926 Found symbol '__ZTSN6google8protobuf10BytesValueE' in multiple binaries: Envoy, Envoy'
05:06:52.048933 Found symbol '__ZN6google8protobuf13MethodOptionsC1ERKS1_' in multiple binaries: Envoy, Envoy'
05:06:52.048940 Found symbol '__ZN6google8protobuf6StructC1ERKS1_' in multiple binaries: Envoy, Envoy'
05:06:52.048993 Found symbol '__ZNK6google8protobuf11EnumOptions39InternalSerializeWithCachedSizesToArrayEPhPNS0_2io19EpsCopyOutputStreamE' in multiple binaries: Envoy, Envoy'
05:06:52.049002 Found symbol '__ZN6google8protobuf36_GeneratedCodeInfo_default_instance_E' in multiple binaries: Envoy, Envoy'
05:06:52.049008 Found symbol '__ZNK6google8protobuf11DoubleValue13SetCachedSizeEi' in multiple binaries: Envoy, Envoy'
05:06:52.049015 Found symbol '__ZN6google8protobuf17FileDescriptorSet14_InternalParseEPKcPNS0_8internal12ParseContextE' in multiple binaries: Envoy, Envoy'
05:06:52.049021 Found symbol '__ZN6google8protobuf12OneofOptionsC2EPNS0_5ArenaE' in multiple binaries: Envoy, Envoy'

These symbols can easily be dumped and searched using:

$ nm -arch arm64 -WUgj /path/to/Envoy.framework/Envoy

I tried going back several versions of Envoy, and found that the commit prior to the last protobuf bump to v3.10.1 did not have duplicate symbols. This leads me to believe that the protobuf bump commit resulted in duplication of these symbols.

Fixing

It's not yet clear to me whether this issue has been fixed by upstream fixes to protobuf, as v3.11.2 is now the latest release. We should try updating to see if this resolves the problem.

Updating protobuf

I started updating protobuf in Envoy, but it appears that a change to protobuf resulted in deprecations being surfaced as warnings, which subsequently causes Envoy Mobile not to compile due to the fact that Envoy uses some of these deprecated entities.

The same change in protobuf that started surfacing these warnings also added -Wno-deprecated-declarations, effectively disabling the warnings in the protobuf library itself. Thus, we can either:

1. Silence these warnings in Envoy
2. Investigate what it'll take to migrate off of the deprecated entities

[junr03]

I wanted to surface another linking issue in iOS that @htuch and I discussed on an envoy PR: envoyproxy/envoy#9191

[keith]

Filed some upstream issues that I believe to be the cause of this:

protocolbuffers/protobuf#7046
bazelbuild/bazel#10484

[rebello95]

Thanks so much @keith.

There's a fix that has been submitted to fix this issue in Bazel, after which we should hopefully be able to bump Bazel and update: bazelbuild/bazel#10493