Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set names to various CMSSW threads #44912

Open
makortel opened this issue May 6, 2024 · 11 comments
Open

Set names to various CMSSW threads #44912

makortel opened this issue May 6, 2024 · 11 comments

Comments

@makortel
Copy link
Contributor

makortel commented May 6, 2024

Setting names like TBB pool or crash monitor to the various threads (with pthread_setname_np(), as suggested in #44901 (comment)) created by cmsRun could be useful for interpreting stack traces on a process with many thread.

One detail that I noticed in #44901 (comment) is that the name of a thread gets forwarded to all threads created by that thread. This detail may be mostly relevant for the case where edm::WaitingTaskWithArenaHolder is used on a job that was configured to use 1 TBB thread. In that case the onetbb::task_arena::enqueue() call in edm::WaitingTaskWithArenaHolder::doneWaiting() creates a new TBB thread, that inherits the name from the (non-)TBB thread, and so on.

@makortel
Copy link
Contributor Author

makortel commented May 6, 2024

assign core

@cmsbuild
Copy link
Contributor

cmsbuild commented May 6, 2024

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

@cmsbuild
Copy link
Contributor

cmsbuild commented May 6, 2024

cms-bot internal usage

@cmsbuild
Copy link
Contributor

cmsbuild commented May 6, 2024

A new Issue was created by @makortel.

@rappoccio, @antoniovilela, @smuzaffar, @sextonkennedy, @makortel, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor Author

makortel commented May 6, 2024

@fwyzard I have a vague recollection you would have brought this up at some point in the past.

@makortel
Copy link
Contributor Author

makortel commented May 6, 2024

Tagging @dan131riley too

@makortel
Copy link
Contributor Author

makortel commented May 6, 2024

For TBB threads, based on oneapi-src/oneTBB#831 and oneapi-src/oneTBB#896 the recommendation from TBB developers for this kind of "do something upon thread creation" seems to be the use of task_scheduler_observer and a thread-local flag to tell if the "initialization" is done already or not. With the observer we could distinguish the main thread from the worker threads (I'm not really sure if that would be useful or not).

The stack trace helper thread in InitRootHandlers would be easy to handle.

The master thread and worker thread pool that we use to run Geant4 work would be easy to handle too.

I'd like to set the names of the Eigen thread pool Tensorflow creates, but I don't know on the top of my head how to do that. Out of the box I'd expect those threads to inherit the name of the TBB threads.

@fwyzard
Copy link
Contributor

fwyzard commented May 6, 2024

Yes - for example, we see the CUDA threads in a GDB dump, and it would be nice to see explicitly

  • the threads used by the DAQ source
  • the threads used by TensorFlow or Eigen
  • the threads used by ROOT, if any
  • etc.

@fwyzard
Copy link
Contributor

fwyzard commented May 6, 2024

I'd like to set the names of the Eigen thread pool Tensorflow creates, but I don't know on the top of my head how to do that. Out of the box I'd expect those threads to inherit the name of the TBB threads.

Do we know when/how they are created ?

If we do, a non-invasive way could be

  • set the name of the current thread
  • let it spawn the Eigen or TF threads, so the inherit the intended name
  • rename back the current thread

@fwyzard
Copy link
Contributor

fwyzard commented May 6, 2024

Or, at least for Eigen, we can hack it and set the names there :-)

@makortel
Copy link
Contributor Author

makortel commented May 6, 2024

Do we know when/how they [Eigen threads] are created ?

I'd guess the first use of Tensorflow would create the threads, I don't really know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants