Support debugging multithreaded programs until they create threads (#…

…10594)

ocamldebug has never worked with multithreaded programs: multiple
threads mess up the communication protocol with the debugger, and the
fork()-based checkpointing mechanism breaks too.

Up to 4.11, the debugger would work fine until the first thread was
created.  Then, bizarre errors would be reported.

In 4.12 (commit e678885), a check was
added, causing any program linked with the threads library to report a
fatal error at start-up time when run under the debugger.

As reported in #10517, this check is too strong: some programs just
happen to be linked with the threads library, e.g. because they use
LWT, but may never start a thread.  There's no reason to refuse to
debug these programs.

This commit reverts the 4.12 change and adds a check in Thread.create
that aborts the program if it is run under the debugger.  This way,
programs linked with the threads library can still be debugged until
the point where they create new threads.

Changes

@@ -68,6 +68,13 @@ Working version
 - #10565: Toplevel value printing: truncate strings only after 8 bytes.
   (Wiktor Kuchta, review by Xavier Leroy)
 
+### Debugging:
+
+- #10517, #10594: when running ocamldebug on a program linked with the
+  threads library, don't fail immediately; instead, allow debugging
+  until the program creates a thread for the first time, then fail cleanly.
+  (Xavier Leroy, report by @anentropic, review by Gabriel Scherer)
+
 ### Manual and documentation:
 
 - #7812, #10475: reworded the description of the behaviors of

otherlibs/systhreads/st_stubs.c

@@ -19,6 +19,7 @@
 #include "caml/backtrace.h"
 #include "caml/callback.h"
 #include "caml/custom.h"
+#include "caml/debugger.h"
 #include "caml/domain.h"
 #include "caml/fail.h"
 #include "caml/io.h"
@@ -558,6 +559,10 @@ CAMLprim value caml_thread_new(value clos)          /* ML */
   caml_thread_t th;
   st_retcode err;
 
+#ifndef NATIVE_CODE
+  if (caml_debugger_in_use)
+    caml_fatal_error("ocamldebug does not support multithreaded programs");
+#endif
   /* Create a thread info block */
   th = caml_thread_new_info();
   if (th == NULL) caml_raise_out_of_memory();

runtime/debugger.c

@@ -137,11 +137,9 @@ static void open_connection(void)
   dbg_in = caml_open_descriptor_in(dbg_socket);
   dbg_out = caml_open_descriptor_out(dbg_socket);
   /* The code in this file does not bracket channel I/O operations with
-     Lock and Unlock, so fail if those are not no-ops. */
-  if (caml_channel_mutex_lock != NULL ||
-      caml_channel_mutex_unlock != NULL ||
-      caml_channel_mutex_unlock_exn != NULL)
-    caml_fatal_error("debugger does not support channel locks");
+     Lock and Unlock, but this is safe because the debugger only works
+     with single-threaded programs.  The program being debugged
+     will abort when it creates a thread. */
   if (!caml_debugger_in_use) caml_putword(dbg_out, -1); /* first connection */
 #ifdef _WIN32
   caml_putword(dbg_out, _getpid());