Crash when running CoreFoundation applications (and wqthread-related problems) #4

Qix- · 2020-02-23T13:32:07Z

I'm trying really hard to debug this but getting nowhere, even with lldb, vgdb and valgrind --vgdb=yes.

The new valgrind patch for MacOS mojave results in a successful install but on a few non-trivial applications Memcheck is causing a SIGILL to be raised. The application in question runs fine, otherwise.

Here is the stack:

==47185== valgrind: Unrecognised instruction at address 0x108afc25c.
==47185==    at 0x108AFC25C: ??? (in /usr/lib/system/libdispatch.dylib)
==47185==    by 0x108AFBD3A: ??? (in /usr/lib/system/libdispatch.dylib)
==47185==    by 0x108AFB2B1: ??? (in /usr/lib/system/libdispatch.dylib)
==47185==    by 0x108AF2720: ??? (in /usr/lib/system/libdispatch.dylib)
==47185==    by 0x108AF267C: ??? (in /usr/lib/system/libdispatch.dylib)
==47185==    by 0x108AE6B9A: ??? (in /usr/lib/system/libdispatch.dylib)
==47185==    by 0x108DD5F3B: ??? (in /usr/lib/system/libsystem_notify.dylib)
==47185==    by 0x108DD4B77: ??? (in /usr/lib/system/libsystem_notify.dylib)
==47185==    by 0x108DD2F6E: ??? (in /usr/lib/system/libsystem_notify.dylib)
==47185==    by 0x104EF87B9: _CFPrefsExtractQuadrupleFromPathIfPossible (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==47185==    by 0x108AE163C: ??? (in /usr/lib/system/libdispatch.dylib)
==47185==    by 0x108AE2D4A: ??? (in /usr/lib/system/libdispatch.dylib)
==47185== Your program just tried to execute an instruction that Valgrind
==47185== did not recognise.  There are two possible reasons for this.
==47185== 1. Your program has a bug and erroneously jumped to a non-code
==47185==    location.  If you are running Memcheck and you just saw a
==47185==    warning about a bad jump, it's probably your program's fault.
==47185== 2. The instruction is legitimate but Valgrind doesn't handle it,
==47185==    i.e. it's Valgrind's fault.  If you think this is the case or
==47185==    you are not sure, please let us know and we'll try to fix it.
==47185== Either way, Valgrind will now raise a SIGILL signal which will
==47185== probably kill your program.

The only other output comes before the application even starts - I don't know if it's relevant or not:

--47185-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option
--47185-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 2 times)
--47185-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 4 times)

I'm having issues creating a minimal reproduction case, as I have no idea which part of the application is causing this to happen (since the stacktrace doesn't give me any application-specific information).

Any tips on how to debug this would be appreciated :)

The text was updated successfully, but these errors were encountered:

LouisBrunner · 2020-02-23T13:59:45Z

@Qix-, thanks for your report!

The UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option messages are not relevant for this problem (but we should fix them sooner or later...).

Without a program to test, it will be extremely tricky for me to debug. As _CFPrefsExtractQuadrupleFromPathIfPossible appears in the stacktrace, CoreFoundation is involved which means we might be able to reproduce it on non-CLI programs (Safari or else maybe?).

As this is a really similar case to the ptr_munge bug of macOS 10.15 (where valgrind was not setting up the binary properly and hitting a ud2 instruction added by some kind of ASSERT), it might give us a bit more information. Could you include the full input if you ran valgrind with -d -v -v -v -v -v --trace-syscalls=yes --trace-flags=11111111 --trace-children=yes?

LouisBrunner · 2020-02-23T14:30:49Z

I actually managed to reproduce the problem fairly easily, running valgrind with Hex Fiend (I am guessing any .app will work).

I have good and bad news: I already have a fix which I recovered from an old patch (you can try it here), however you will probably run into another problem rightaway: SIGSEGV on start_wqthread.

I also have a fix for that issue, but it is really experimental (see here) and need polishing.

Qix- · 2020-02-23T14:41:18Z

Yep definitely using CoreFoundation (it's a game engine, with a bunch of window calls). Glad it wasn't localized to my application ^^

As for testing the patches, should I rebase one onto the other in order to test both, or does the last link include them both? And should I build via the makefiles or is there a fancy brew command to do so?

LouisBrunner · 2020-02-23T15:04:43Z

I just merged kevent_id because it was done and now it's obvious that it's needed. Which means you could use the feature/wqthread_fix branch directly (keeping in mind that it will probably crash).

Unfortunately, I am not aware of any brew command that does that, you will need to build via the Makefile.

Qix- · 2020-02-23T16:07:08Z

Output from feature/wqthread_fix. Built valgrind with ./autogen.sh && ./configure && make && sudo make install && valgrind /path/to/my/app

--77136-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 8 times)

valgrind: m_syswrap/syswrap-amd64-darwin.c:512 (void wqthread_hijack(Addr, Addr, Addr, Addr, Int, Addr)): Assertion 'tst->os_state.pthread - magic_delta == self' failed.

host stacktrace:
==77136==    at 0x2580521E9: ???

sched status:
  running_tid=0

Thread 1: status = VgTs_WaitSys syscall unix:266 (lwpid 771)
==77136==    at 0x108DF895E: ??? (in /usr/lib/system/libsystem_kernel.dylib)
==77136==    by 0x104F016A6: -[CFPrefsPlistSource handleReply:toRequestNewDataMessage:onConnection:retryCount:error:] (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104F011DA: -[CFPrefsPlistSource handleReply:toRequestNewDataMessage:onConnection:retryCount:error:] (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104F00CF8: -[CFPrefsSearchListSource handleReply:toRequestNewDataMessage:onConnection:retryCount:error:] (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x108EABAB9: ??? (in /usr/lib/system/libxpc.dylib)
==77136==    by 0x104F00C3A: __34-[_CFXPreferences canLookUpAgents]_block_invoke (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104F00AFE: ___CFPrefsDirectMode_block_invoke (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104F00527: ___CFGetCachedUnsandboxedHomeDirectoryForUser_block_invoke (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104F004FD: ___CFGetCachedUnsandboxedHomeDirectoryForUser_block_invoke (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x108AEB671: ??? (in /usr/lib/system/libdispatch.dylib)
==77136==    by 0x108AFBA42: ??? (in /usr/lib/system/libdispatch.dylib)
==77136==    by 0x108AFB595: ??? (in /usr/lib/system/libdispatch.dylib)
==77136==    by 0x10504622A: __CFStringEncodingICUToBytes (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104F8354C: __64-[_CFXPreferences copyKeyListForIdentifier:user:host:container:]_block_invoke (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104EFF40E: -[__NSArrayM getObjects:range:] (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104EFEE41: -[CFPrefsSearchListSource alreadylocked_copyValueForKey:] (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104EFEA08: _CFStringCheckAndGetCharacters (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104EFE950: CFStringHashISOLatin1CString (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104EFE90E: CFStringHashISOLatin1CString (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104EE89B5: -[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:] (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104EE8677: -[_CFXPreferencesHandle copyPrefs] (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104EE83C8: ___CFPrefsCopyDefaultPreferences_block_invoke (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104EE7F88: CFArrayGetCount (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x102AB29ED: -[NSUserDefaults(NSUserDefaults) init] (in /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation)
==77136==    by 0x102ABAB4F: +[NSUserDefaults(NSUserDefaults) standardUserDefaults] (in /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation)
==77136==    by 0x103760C77: +[NSApplication initialize] (in /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit)
==77136==    by 0x1066AE4B1: ??? (in /usr/lib/libobjc.A.dylib)
==77136==    by 0x1066AE864: ??? (in /usr/lib/libobjc.A.dylib)
==77136==    by 0x1066AE79A: ??? (in /usr/lib/libobjc.A.dylib)
==77136==    by 0x1066AF62E: ??? (in /usr/lib/libobjc.A.dylib)
==77136==    by 0x10669E68F: ??? (in /usr/lib/libobjc.A.dylib)
==77136==    by 0x10669E113: ??? (in /usr/lib/libobjc.A.dylib)
==77136==    by 0x10056E8E8: Cocoa_RegisterApp (SDL_cocoaevents.m:404)
==77136==    by 0x10057603B: Cocoa_CreateDevice (SDL_cocoavideo.m:58)
==77136==    by 0x100487B3A: SDL_VideoInit_REAL (SDL_video.c:505)
==77136==    by 0x10037D379: SDL_InitSubSystem_REAL (SDL.c:206)
==77136==    by 0x10037D712: SDL_Init_REAL (SDL.c:291)
==77136==    by 0x1003953E6: SDL_Init (SDL_dynapi_procs.h:85)
==77136==    by 0x100221249: tide::renderer::init_render_thread() (renderer.cc:79)
==77136==    by 0x1001BD0E3: main (main.cc:60)
client stack range: [0x105E97000 0x106696FFF] client SP: 0x106694E28
valgrind stack range: [0x700001AA0000 0x700001B9FFFF] top usage: 9800 of 1048576

Thread 2: status = VgTs_WaitSys syscall unix:368 (lwpid 5387)
==77136==    at 0x108DF8BFA: ??? (in /usr/lib/system/libsystem_kernel.dylib)
==77136==    by 0x108E4D6E5: ??? (in /usr/lib/system/libsystem_pthread.dylib)
==77136==    by 0x108E4D3FC: ??? (in /usr/lib/system/libsystem_pthread.dylib)
client stack range: ??????? client SP: 0x70000DDE6F98
valgrind stack range: [0x700006142000 0x700006241FFF] top usage: 3304 of 1048576

Thread 3: status = VgTs_WaitSys syscall mach:31 (lwpid 9987)
==77136==    at 0x108DF721A: ??? (in /usr/lib/system/libsystem_kernel.dylib)
==77136==    by 0x108DF7767: ??? (in /usr/lib/system/libsystem_kernel.dylib)
==77136==    by 0x108EA60D7: ??? (in /usr/lib/system/libxpc.dylib)
==77136==    by 0x108EA5E30: ??? (in /usr/lib/system/libxpc.dylib)
==77136==    by 0x108EB59F2: ??? (in /usr/lib/system/libxpc.dylib)
==77136==    by 0x108EAA1C6: ??? (in /usr/lib/system/libxpc.dylib)
==77136==    by 0x108EA9D93: ??? (in /usr/lib/system/libxpc.dylib)
==77136==    by 0x108EA9BAB: ??? (in /usr/lib/system/libxpc.dylib)
==77136==    by 0x108EA9B36: ??? (in /usr/lib/system/libxpc.dylib)
==77136==    by 0x104F00707: -[_CFXPreferences withConnectionForRole:performBlock:] (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104F0051A: ___CFGetCachedUnsandboxedHomeDirectoryForUser_block_invoke (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x104F004FD: ___CFGetCachedUnsandboxedHomeDirectoryForUser_block_invoke (in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation)
==77136==    by 0x108AEB671: ??? (in /usr/lib/system/libdispatch.dylib)
==77136==    by 0x108AFAF94: ??? (in /usr/lib/system/libdispatch.dylib)
==77136==    by 0x108AEB63C: ??? (in /usr/lib/system/libdispatch.dylib)
==77136==    by 0x108AF9508: ??? (in /usr/lib/system/libdispatch.dylib)
==77136==    by 0x108AF9B45: ??? (in /usr/lib/system/libdispatch.dylib)
==77136==    by 0x108E4D6B2: ??? (in /usr/lib/system/libsystem_pthread.dylib)
==77136==    by 0x108E4D3FC: ??? (in /usr/lib/system/libsystem_pthread.dylib)
client stack range: ??????? client SP: 0x70000DE69B28
valgrind stack range: [0x700006246000 0x700006345FFF] top usage: 3872 of 1048576

If you'd like, I can do a run with the high-verbosity flags you mentioned before.

LouisBrunner · 2020-02-23T16:17:47Z

No it's alright, I get the exact same error, so at least there is comfort in that...

I'll need to continue working on the wqthread fix as it's obviously still buggy.

Qix- · 2020-02-23T16:25:22Z

Alright, sounds good :) Let me know if I can help in any way! Thanks for all of the work you've done, it's incredibly appreciated.

LouisBrunner · 2020-03-05T22:51:47Z

Thanks a lot for your kind words! I will try to look into it, but if you want to investigate yourself, it would be greatly appreciated.

ernesernesto · 2021-06-27T10:41:14Z

I'm not sure If this is the right place to ask this, is there any tool that can be used to detect memory leak beside valgrind? Because of wqthread issues it's impossible to check multithread app with valgrind in osx according to this https://bugs.kde.org/show_bug.cgi?id=380269 which might be related, link is already 4 years old and I guess it's safe to assume it would not be fixed shortly

LouisBrunner added the bug Something isn't working label Feb 23, 2020

LouisBrunner changed the title ~~Memcheck causing segfault in otherwise working application~~ Crash when running CoreFoundation applications (and wqthread-related problems) Feb 23, 2020

LouisBrunner pinned this issue Feb 23, 2020

LouisBrunner added the help wanted Extra attention is needed label Mar 5, 2020

LouisBrunner mentioned this issue Apr 22, 2020

Missing symbols for _mig_strncpy[_zerofill] #5

Closed

LouisBrunner added this to To do in Roadmap May 4, 2020

LouisBrunner moved this from To do to In progress in Roadmap May 6, 2020

TheButlah mentioned this issue Jun 25, 2020

Crash when importing opencv in python #13

Closed

LouisBrunner mentioned this issue Jan 8, 2021

Valgrind segfaults running threaded code under MacOS 10.14.6 #25

Closed

LouisBrunner moved this from In progress to To do in Roadmap Apr 18, 2021

LouisBrunner mentioned this issue May 6, 2023

Getting seg fault from valgrind #83

Open

LouisBrunner mentioned this issue Jul 26, 2023

Running into error "UNKNOWN workq_ops option 1024" when running #95

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash when running CoreFoundation applications (and wqthread-related problems) #4

Crash when running CoreFoundation applications (and wqthread-related problems) #4

Qix- commented Feb 23, 2020 •

edited

LouisBrunner commented Feb 23, 2020

LouisBrunner commented Feb 23, 2020

Qix- commented Feb 23, 2020

LouisBrunner commented Feb 23, 2020

Qix- commented Feb 23, 2020

LouisBrunner commented Feb 23, 2020

Qix- commented Feb 23, 2020

LouisBrunner commented Mar 5, 2020

ernesernesto commented Jun 27, 2021

Crash when running CoreFoundation applications (and wqthread-related problems) #4

Crash when running CoreFoundation applications (and wqthread-related problems) #4

Comments

Qix- commented Feb 23, 2020 • edited

LouisBrunner commented Feb 23, 2020

LouisBrunner commented Feb 23, 2020

Qix- commented Feb 23, 2020

LouisBrunner commented Feb 23, 2020

Qix- commented Feb 23, 2020

LouisBrunner commented Feb 23, 2020

Qix- commented Feb 23, 2020

LouisBrunner commented Mar 5, 2020

ernesernesto commented Jun 27, 2021

Qix- commented Feb 23, 2020 •

edited