Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Virtual thread metrics #3956

Open
shakuzen opened this issue Jul 5, 2023 · 9 comments · May be fixed by #5067
Open

Virtual thread metrics #3956

shakuzen opened this issue Jul 5, 2023 · 9 comments · May be fixed by #5067
Labels
enhancement A general enhancement module: micrometer-core An issue that is related to our core module
Milestone

Comments

@shakuzen
Copy link
Member

shakuzen commented Jul 5, 2023

Follow-up from #3650. See https://openjdk.org/jeps/444#JDK-Flight-Recorder-JFR

Existing thread metrics do not cover virtual threads - ThreadMXBean only provides statistics on platform threads.
New potential instrumentation of virtual threads is only available via JFR. We can count started/finished virtual threads. The other events available are:

  • jdk.VirtualThreadPinned indicates that a virtual thread was pinned (and its carrier thread wasn’t freed). This event is enabled by default with a threshold of 20 ms.
  • jdk.VirtualThreadSubmitFailed indicates that starting or unparking a virtual thread failed, probably due to a resource issue. Parking a virtual thread releases the underlying carrier thread to do other work, and unparking a virtual thread schedules it to continue. This event is enabled by default.
@shakuzen shakuzen added enhancement A general enhancement module: micrometer-core An issue that is related to our core module labels Jul 5, 2023
@shakuzen shakuzen added this to the 1.12 backlog milestone Jul 5, 2023
@shakuzen shakuzen modified the milestones: 1.12 backlog, 1.next Oct 26, 2023
@juliojgd
Copy link

Any developments here?

@jonatan-ivanov
Copy link
Member

Not much I guess. Since this is not available through JMX (like metrics for platform threads are), we need to utilize JFR and jdk.VirtualThreadStart/jdk.VirtualThreadEnd events (disabled by default) which might be a bit more effort. Are you interested in submitting a PR?

@juliojgd
Copy link

juliojgd commented Feb 7, 2024

I am not very familiar with JFR but, in order to retrieve the events count, shouldn't we start and stop a JFR recording? Or even keeping an active recording for the whole life of the JVM?

My knowledge about JFR is not enough to contribute this PR. Anyway I will try to find if in Java 22 or beyond there is any plan or JEP to enhance ThreadMXBean (or create a specific one for VT's) that could ease the retrieval of this measures.

@jonatan-ivanov
Copy link
Member

I am not very familiar with JFR but, in order to retrieve the events count, shouldn't we start and stop a JFR recording?

I need to look into this but not necessarily in the classic sense where JFR is writing the data into a file. What we need is receiving the JFR events, we need to investigate how to do it so that it does not disturb the app.

Anyway I will try to find if in Java 22 or beyond there is any plan or JEP to enhance ThreadMXBean (or create a specific one for VT's) that could ease the retrieval of this measures.

Having this info coming from ThreadMXBean would be the best since we are already interacting with it for platform threads.

@ArtyomGabeev
Copy link
Contributor

Hello here,

Let me know if you want me try and contribute support of VT metrics based on JFR continuous event listener.

Or are we waiting this data to be exposed via jmx?

@shakuzen
Copy link
Member Author

We would consider an implementation based on the JFR events. I am not aware of any plans for anything to be added to JMX, so I don't think just waiting is necessarily likely to yield results. We could ask on the corresponding JDK mailing list about adding support for virtual thread metrics to JMX if there's a reason that's better over the JFR event route.

It would also be good for users to share what virtual thread metrics in particular they are interested in and why so we can understand what will be most helpful and why.

@ArtyomGabeev
Copy link
Contributor

ArtyomGabeev commented May 12, 2024

I think we may be interested in two JFR events:

  1. VirtualThreadPinnedEvent - expose as a time, with pinning duration
  2. VirtualThreadSubmitFailedEvent - expose as a counter

For the first one, things to consider:

  1. Add a tag with stackframe where pinning happened? e.g.: com.test.ThreadPinningExample.sleep(Duration)
  2. Additionally, I would like to be able to perform log once with extended strack trace, but I'm not sure this is a responsibility of micrometer library.

Here is an example of VirtualThreadPinnedEvent:

jdk.VirtualThreadPinned {
  startTime = 16:58:30.051 (2024-04-14)
  duration = 251 ms
  eventThread = "web-vt-729a03c1-5456-4686-90b2-b2659291752f" (javaThreadId = 69, virtual)
  stackTrace = [
    java.lang.VirtualThread.parkOnCarrierThread(boolean, long) line: 677
    java.lang.VirtualThread.parkNanos(long) line: 636
    java.lang.VirtualThread.sleepNanos(long) line: 793
    java.lang.Thread.sleep(long) line: 507
    com.test.ThreadPinningExample.sleep(Duration) line: 36
    ...
  ]
}

@ArtyomGabeev ArtyomGabeev linked a pull request May 12, 2024 that will close this issue
@ArtyomGabeev
Copy link
Contributor

ArtyomGabeev commented May 12, 2024

I've opened draft PR to demonstrate intentions and come up with the question:

I think this instrumentation should be specific to java-21. Should I go and create a java-21 source set or separate gradle module?

@ArtyomGabeev
Copy link
Contributor

I'm not sure we need to process jdk.VirtualThreadStart and jdk.VirtualThreadEnd events.
E.g. in virtual thread per model request, amount created events equals to rps, which can be achieved via other ways.
If virtual threads used as part of executor, there is already exist intsrumentation - ExecutorServiceMetrics

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement A general enhancement module: micrometer-core An issue that is related to our core module
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants