Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v11.0.0 - very frequent segfault errors - making this gem unusable #339

Closed
vijuSR opened this issue Jul 30, 2021 · 7 comments
Closed

v11.0.0 - very frequent segfault errors - making this gem unusable #339

vijuSR opened this issue Jul 30, 2021 · 7 comments

Comments

@vijuSR
Copy link

vijuSR commented Jul 30, 2021 •
edited

We are running into 2 types of segfaults, one seems to be intermittent (similar to #324 (comment)), the other is very frequent. Trace info for the frequent one failure.log

We had to upgrade, before API v6 sunset https://developers.google.com/google-ads/api/docs/sunset-dates, and to do that we had to upgrade its dependencies on google-protobuf to v3.17.3 and grpc to v1.38.0, either of which seems to be the root cause here.

But the point is, this error is making the google-ads gem unusable and we are left with no other option. . Please have a look into this ASAP.
@mcloonan
Copy link
Member

I'm reaching out internally to the protobuf team to see if there's something they're aware of. In the meanwhile, can you provide any details about what you're trying to do when the frequent seg fault occurs? That might help us narrow it down faster.

@vijuSR
Copy link
Author

vijuSR commented Jul 30, 2021 •
edited

@mcloonan thanks for prompt response.

We found the reason for the frequent segfaults, it's an issue with the grpc+jemalloc grpc/grpc#25450. The workaround mentioned on the issue thread seems to be working for now.

But still, we don't have an answer to the intermittent segfaults which appear to be happening during a protobuf op in the google ads service. Stack trace info:

/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/resources/campaign_criterion_simulation_pb.rb:14: [BUG] Segmentation fault at 0x0000000000000048
--
ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
 
-- Control frame information -----------------------------------------------
c:0050 p:---- s:0279 e:000278 CFUNC  :add_message
c:0049 p:0008 s:0274 E:0000d0 BLOCK  /home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/resources/campaign_criterion_sim [FINISH]
c:0048 p:---- s:0271 e:000270 CFUNC  :instance_eval
c:0047 p:---- s:0268 e:000267 CFUNC  :add_file
c:0046 p:0010 s:0262 E:000d40 BLOCK  /home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/resources/campaign_criterion_sim [FINISH]
c:0045 p:---- s:0259 e:000258 CFUNC  :instance_eval
c:0044 p:---- s:0256 e:000255 CFUNC  :build
c:0043 p:0067 s:0252 E:000320 TOP    /home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/resources/campaign_criterion_sim [FINISH]
c:0042 p:---- s:0249 e:000248 CFUNC  :require
c:0041 p:0006 s:0244 e:000243 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/elastic-apm-2.6.0/lib/elastic_apm/spies.rb:74
c:0040 p:0054 s:0237 e:000236 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23
c:0039 p:0008 s:0231 e:000230 BLOCK  /home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324
c:0038 p:0068 s:0228 e:000227 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:291
c:0037 p:0011 s:0221 e:000220 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324
c:0036 p:0244 s:0215 e:000214 TOP    /home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/services/google_ads_service_pb.r [FINISH]
c:0035 p:---- s:0212 e:000211 CFUNC  :require
c:0034 p:0006 s:0207 e:000206 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/elastic-apm-2.6.0/lib/elastic_apm/spies.rb:74
c:0033 p:0054 s:0200 e:000199 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23
c:0032 p:0008 s:0194 e:000193 BLOCK  /home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324
c:0031 p:0068 s:0191 e:000190 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:291
c:0030 p:0011 s:0184 e:000183 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324
c:0029 p:0006 s:0178 e:000177 TOP    /home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/services/google_ads_service/clie [FINISH]
c:0028 p:---- s:0175 e:000174 CFUNC  :require
c:0027 p:0006 s:0170 e:000169 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/elastic-apm-2.6.0/lib/elastic_apm/spies.rb:74
c:0026 p:0054 s:0163 e:000162 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23
c:0025 p:0008 s:0157 e:000156 BLOCK  /home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324
c:0024 p:0068 s:0154 e:000153 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:291
c:0023 p:0011 s:0147 e:000146 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324
c:0022 p:0048 s:0141 e:000140 TOP    /home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/services/google_ads_service.rb:2 [FINISH]
c:0021 p:---- s:0137 e:000136 CFUNC  :require
c:0020 p:0006 s:0132 e:000131 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/elastic-apm-2.6.0/lib/elastic_apm/spies.rb:74
c:0019 p:0054 s:0125 e:000124 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23
c:0018 p:0008 s:0119 e:000118 BLOCK  /home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324
c:0017 p:0068 s:0116 e:000115 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:291
c:0016 p:0011 s:0109 e:000108 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324
c:0015 p:0006 s:0103 e:000102 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/factories/v6/services.rb:2047
c:0014 p:0040 s:0097 e:000096 METHOD /home/deploy/.rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/delegate.rb:83
c:0013 p:0009 s:0088 e:000087 METHOD /home/deploy/clarisights/current/app/adwords/ads_api.rb:421
c:0012 p:0005 s:0084 e:000083 BLOCK  /home/deploy/clarisights/current/app/adwords/ads_api.rb:143
c:0011 p:0020 s:0080 e:000079 BLOCK  /home/deploy/clarisights/current/lib/retry.rb:34
c:0010 p:0206 s:0075 e:000074 METHOD /home/deploy/.rvm/gems/ruby-2.6.5/gems/retryable-3.0.5/lib/retryable.rb:72
c:0009 p:0129 s:0060 E:001748 METHOD /home/deploy/clarisights/current/lib/retry.rb:32
c:0008 p:0058 s:0051 E:002708 METHOD /home/deploy/clarisights/current/app/adwords/ads_api.rb:142
c:0007 p:0147 s:0044 e:000043 METHOD /home/deploy/clarisights/current/app/adwords/ads_api.rb:413
c:0006 p:0014 s:0036 e:000035 BLOCK  /home/deploy/clarisights/current/app/models/adwords_ad_account.rb:1742
c:0005 p:0023 s:0033 e:000032 METHOD /home/deploy/.rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/benchmark.rb:308
c:0004 p:0014 s:0028 e:000027 METHOD /home/deploy/clarisights/current/lib/ruby_util.rb:10
c:0003 p:0015 s:0023 e:000022 BLOCK  /home/deploy/clarisights/current/app/models/adwords_ad_account.rb:1741
c:0002 p:0151 s:0017 e:000016 BLOCK  /home/deploy/clarisights/current/lib/background_threads.rb:536 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]
 
-- Ruby level backtrace information ----------------------------------------
/home/deploy/clarisights/current/lib/background_threads.rb:536:in `block in worker_thread'
/home/deploy/clarisights/current/app/models/adwords_ad_account.rb:1741:in `block (2 levels) in update_additional_campaign_reports_from_ads_api'
/home/deploy/clarisights/current/lib/ruby_util.rb:10:in `execution_time'
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/benchmark.rb:308:in `realtime'
/home/deploy/clarisights/current/app/models/adwords_ad_account.rb:1742:in `block (3 levels) in update_additional_campaign_reports_from_ads_api'
/home/deploy/clarisights/current/app/adwords/ads_api.rb:413:in `get_sub_channel_campaign_reports'
/home/deploy/clarisights/current/app/adwords/ads_api.rb:142:in `search'
/home/deploy/clarisights/current/lib/retry.rb:32:in `perform'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/retryable-3.0.5/lib/retryable.rb:72:in `retryable'
/home/deploy/clarisights/current/lib/retry.rb:34:in `block in perform'
/home/deploy/clarisights/current/app/adwords/ads_api.rb:143:in `block in search'
/home/deploy/clarisights/current/app/adwords/ads_api.rb:421:in `main_service'
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/delegate.rb:83:in `method_missing'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/factories/v6/services.rb:2047:in `google_ads'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:291:in `load_dependency'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324:in `block in require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/elastic-apm-2.6.0/lib/elastic_apm/spies.rb:74:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/elastic-apm-2.6.0/lib/elastic_apm/spies.rb:74:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/services/google_ads_service.rb:27:in `<top (required)>'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:291:in `load_dependency'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324:in `block in require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/elastic-apm-2.6.0/lib/elastic_apm/spies.rb:74:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/elastic-apm-2.6.0/lib/elastic_apm/spies.rb:74:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/services/google_ads_service/client.rb:20:in `<top (required)>'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:291:in `load_dependency'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324:in `block in require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/elastic-apm-2.6.0/lib/elastic_apm/spies.rb:74:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/elastic-apm-2.6.0/lib/elastic_apm/spies.rb:74:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/services/google_ads_service_pb.rb:39:in `<top (required)>'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:291:in `load_dependency'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324:in `block in require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/zeitwerk-2.3.0/lib/zeitwerk/kernel.rb:23:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/elastic-apm-2.6.0/lib/elastic_apm/spies.rb:74:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/elastic-apm-2.6.0/lib/elastic_apm/spies.rb:74:in `require'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/resources/campaign_criterion_simulation_pb.rb:12:in `<top (required)>'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/resources/campaign_criterion_simulation_pb.rb:12:in `build'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/resources/campaign_criterion_simulation_pb.rb:12:in `instance_eval'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/resources/campaign_criterion_simulation_pb.rb:13:in `block in <top (required)>'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/resources/campaign_criterion_simulation_pb.rb:13:in `add_file'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/resources/campaign_criterion_simulation_pb.rb:13:in `instance_eval'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/resources/campaign_criterion_simulation_pb.rb:14:in `block (2 levels) in <top (required)>'
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/resources/campaign_criterion_simulation_pb.rb:14:in `add_message'
 
-- Machine register context ------------------------------------------------
RIP: 0x00007f647137595b RBP: 0x00007f634d9d5930 RSP: 0x00007f634d9d5890
RAX: 0x00007f6389285ad8 RBX: 0x00007f63892490a0 RCX: 0x00007f6389283480
RDX: 0x0000000000000028 RDI: 0x00007f6389285ad0 RSI: 0x0000000000000000
R8: 0x00007f6389283480  R9: 0x00000000000000fd R10: 0x0000564c484d0240
R11: 0x00007f63688f10d0 R12: 0x00007f63688f4238 R13: 0x0000000000000000
R14: 0x0000000000000001 R15: 0x00007f63688f4008 EFL: 0x0000000000010206
 
-- C level backtrace information -------------------------------------------
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_vm_bugreport+0x769) [0x7f64710177e9] vm_dump.c:715
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_bug_context+0xe7) [0x7f6470e5b157] error.c:609
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(sigsegv+0x42) [0x7f6470f7e7f2] signal.c:998
/lib/x86_64-linux-gnu/libc.so.6(0x7f64709ec040) [0x7f64709ec040]
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-protobuf-3.17.3-x86_64-linux/lib/google/2.6/protobuf_c.so(0x7f647137595b) [0x7f647137595b]
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_cfunc+0x10a) [0x7f6470ff69aa] vm_insnhelper.c:1908
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_method+0xe3) [0x7f6471009d23] vm_insnhelper.c:2400
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_exec_core+0x214) [0x7f64710004c4] insns.def:750
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_vm_exec+0xaf) [0x7f6471006dbf] vm.c:1885
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(invoke_block_from_c_bh+0x1f4) [0x7f6471007f44] vm.c:1092
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(yield_under+0x1fd) [0x7f64710082cd] vm.c:1147
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(specific_eval+0x51) [0x7f647100a951] vm_eval.c:1618
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_vm_call0+0x2e6) [0x7f647100b5a6] vm_eval.c:86
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-protobuf-3.17.3-x86_64-linux/lib/google/2.6/protobuf_c.so(0x7f64713756c6) [0x7f64713756c6]
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_cfunc+0x10a) [0x7f6470ff69aa] vm_insnhelper.c:1908
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_method+0xe3) [0x7f6471009d23] vm_insnhelper.c:2400
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_exec_core+0x214) [0x7f64710004c4] insns.def:750
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_vm_exec+0xaf) [0x7f6471006dbf] vm.c:1885
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(invoke_block_from_c_bh+0x1f4) [0x7f6471007f44] vm.c:1092
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(yield_under+0x1fd) [0x7f64710082cd] vm.c:1147
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(specific_eval+0x51) [0x7f647100a951] vm_eval.c:1618
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_vm_call0+0x2e6) [0x7f647100b5a6] vm_eval.c:86
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-protobuf-3.17.3-x86_64-linux/lib/google/2.6/protobuf_c.so(0x7f64713755c6) [0x7f64713755c6]
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_cfunc+0x10a) [0x7f6470ff69aa] vm_insnhelper.c:1908
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_method+0xe3) [0x7f6471009d23] vm_insnhelper.c:2400
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_exec_core+0x214) [0x7f64710004c4] insns.def:750
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_vm_exec+0xaf) [0x7f6471006dbf] vm.c:1885
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(invoke_block_from_c_bh+0x1f4) [0x7f6471007f44] vm.c:1092
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(yield_under+0x1fd) [0x7f64710082cd] vm.c:1147
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(specific_eval+0x51) [0x7f647100a951] vm_eval.c:1618
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_vm_call0+0x2e6) [0x7f647100b5a6] vm_eval.c:86
/home/deploy/.rvm/gems/ruby-2.6.5/gems/google-protobuf-3.17.3-x86_64-linux/lib/google/2.6/protobuf_c.so(0x7f64713755c6) [0x7f64713755c6]
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_cfunc+0x10a) [0x7f6470ff69aa] vm_insnhelper.c:1908
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_method+0xe3) [0x7f6471009d23] vm_insnhelper.c:2400
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_exec_core+0x214) [0x7f64710004c4] insns.def:750
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_vm_exec+0xaf) [0x7f6471006dbf] vm.c:1885
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_load_internal0+0x107) [0x7f6470eb6227] load.c:612
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_require_internal+0xed1) [0x7f6470eb9a91] load.c:1029
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_require_safe+0x9) [0x7f6470eb9cd9] load.c:1075
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_cfunc+0x10a) [0x7f6470ff69aa] vm_insnhelper.c:1908
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_exec_core+0x152) [0x7f6471000402] insns.def:765
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_vm_exec+0xaf) [0x7f6471006dbf] vm.c:1885
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_load_internal0+0x107) [0x7f6470eb6227] load.c:612
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_require_internal+0xed1) [0x7f6470eb9a91] load.c:1029
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_require_safe+0x9) [0x7f6470eb9cd9] load.c:1075
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_cfunc+0x10a) [0x7f6470ff69aa] vm_insnhelper.c:1908
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_method+0xe3) [0x7f6471009d23] vm_insnhelper.c:2400
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_exec_core+0x152) [0x7f6471000402] insns.def:765
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_vm_exec+0xaf) [0x7f6471006dbf] vm.c:1885
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_load_internal0+0x107) [0x7f6470eb6227] load.c:612
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_require_internal+0xed1) [0x7f6470eb9a91] load.c:1029
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_require_safe+0x9) [0x7f6470eb9cd9] load.c:1075
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_cfunc+0x10a) [0x7f6470ff69aa] vm_insnhelper.c:1908
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_method+0xe3) [0x7f6471009d23] vm_insnhelper.c:2400
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_exec_core+0x152) [0x7f6471000402] insns.def:765
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_vm_exec+0xaf) [0x7f6471006dbf] vm.c:1885
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_load_internal0+0x107) [0x7f6470eb6227] load.c:612
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_require_internal+0xed1) [0x7f6470eb9a91] load.c:1029
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_require_safe+0x9) [0x7f6470eb9cd9] load.c:1075
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_cfunc+0x10a) [0x7f6470ff69aa] vm_insnhelper.c:1908
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_call_method+0xe3) [0x7f6471009d23] vm_insnhelper.c:2400
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(vm_exec_core+0x152) [0x7f6471000402] insns.def:765
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_vm_exec+0xaf) [0x7f6471006dbf] vm.c:1885
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(rb_vm_invoke_proc+0x227) [0x7f647100a557] vm.c:1092
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(thread_do_start+0x250) [0x7f6470fbf670] thread.c:686
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(thread_start_func_2+0x1be) [0x7f6470fc1d1e] thread.c:725
/home/deploy/.rvm/rubies/ruby-2.6.5/lib/libruby.so.2.6(thread_start_func_1+0xcb) [0x7f6470fc22db] thread_pthread.c:1030
/lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb) [0x7f64705786db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f6470ace71f]

@vijuSR
Copy link
Author

vijuSR commented Jul 30, 2021

^ I've corrected the reference to the grpc issue in the above message: grpc/grpc#25450

@haberman haberman mentioned this issue Jul 31, 2021
Closed
@haberman
Copy link

I think this could possibly be related to protocolbuffers/protobuf#8842

That issue was related to the DSL. In protocolbuffers/protobuf#8850 I ported the DSL from C to pure Ruby.

Would you have any easy way of trying out protocolbuffers/protobuf#8850 to see if it fixes these crashes?

@vijuSR
Copy link
Author

vijuSR commented Aug 1, 2021

Thanks @haberman for looking into this. Yes, protocolbuffers/protobuf#8842 seems related.

Would you have any easy way of trying out protocolbuffers/protobuf#8850 to see if it fixes these crashes?

The problem with testing protocolbuffers/protobuf#8850 for the fix is, this segfault is not deterministic. As of now, the probability of it happening in a task is ~0.0015 and they appeared to go away with a retry.

Everytime the error shows up in a random google-ads resource file during a similar protobuf op, examples:

  • google-ads-googleads-8.0.0/lib/google/ads/google_ads/v6/resources/campaign_criterion_simulation_pb.rb:14:in add_message
  • google-ads-googleads-11.0.0/lib/google/ads/google_ads/v7/resources/account_budget_pb.rb:15: add_message

One more observation is, this started after we upgraded:

  • google-protobuf to 3.17.3 (x86_64-linux) from 3.13.0
  • grpc to 1.38.0 (x86_64-linux) from 1.32.0

@haberman
Copy link

haberman commented Aug 1, 2021

add_message was rewritten in protocolbuffers/protobuf#8850 to be implemented in Ruby instead of C, so I think the probability that the PR will fix it is high.

google-protobuf to 3.17.3 (x86_64-linux) from 3.13.0

A large rewrite (protocolbuffers/protobuf#8184) landed in 3.15.0 (https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0), so it may be related to that.

btw that rewrite should also deliver some significant performance improvements, so hopefully you'll see some benefit from that too. :)

@vijuSR
Copy link
Author

vijuSR commented Aug 13, 2021

Thanks guys for looking into this. I'll try this change #339 (comment) in the next google-protobuf release. I'm assuming the suggested (probable) fix will work for the rare segfaults, so, I'm closing this issue now.

@vijuSR vijuSR closed this as completed Aug 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@haberman @mcloonan @vijuSR