In this modern world, thread dumps are still analyzed in a tedious & manual mode i.e., you have to get hold of DevOps team, ask them to send you the thread dumps, then they will mail you the thread dumps, then you will upload the dumps in to a thread dump analysis tool, then you have to apply your intelligence to analyze it. There is no programmatic way to analyze thread dumps in a proactive manner. Thus to eliminate this hassle, fastthread.io is introducing a RESTful API to analyze thread dumps. With one line of CURL command, you can get your thread dumps analyzed instantly.

Here are a few use cases where this API can be extremely useful.

Use case 1: Production Health Check

Most of the DevOps invokes a simple Http ping to check the application’s health. This ping is good to detect whether application is alive or not. But it won’t inform anything beyond that. It won’t report whether application is poorly responding, it won’t report whether Database connections are locked up, it won’t report whether CPU is spiking up, it won’t report whether memory is leaking. Even if it reports those issue, still you would have to manually analyze to see what is causing these problems. Now with the introduction of this Thread dump analysis API, you will not only know the healthiness of the application – but also the root cause the problem. Beauty of this health check is: It doesn’t add any overhead (or negligible overhead). Because entire analysis of the thread dumps are done on a remote server. The only thing you would have to do is capture the thread dumps from your production machines and invoke this API.

Use case 2: Performance Tests

When you conduct performance tests, you might want to take thread dumps on a periodic basis and get it analyzed through the API. In case if thread count goes beyond a threshold or if too many threads are WAITING or if any threads are BLOCKED for a prolonged period or lock isn’t getting released, it needs to get in to the visibility. It should be analyzed before code hits the production. In such circumstance this API will come very handy. Because analyzing several thread dumps meticulously is impossible for anyone.

Use case 3: Continuous Integration

As part of continuous integration it’s highly encouraged to execute performance tests. Thread dumps should be captured and it can be analyzed using the API.  If API reports any problems, then build can be failed. In this way, you can catch the performance degradation right during code commit time instead of catching it in performance labs or production.

How to invoke Thread dump analysis API?

Invoking Thread dump analysis API is very extremely simple:

  1. Register with us. We will email you the API key. This is a one-time setup process. Note: If you have purchased enterprise version, you don’t have to register. API key will be provided to you as part of installation instruction.
  2. Post HTTP request to http://api.fastthread.io/fastthread-api?apiKey={API_KEY_SENT_IN_EMAIL}
  3. The body of the HTTP request should contain the Thread dump that needs to be analyzed. You can either send 1 thread dump or multiple thread dumps in the same request.
  4. HTTP Response will be sent back in JSON format. JSON has several important stats about the Thread dump. Primary element to look in the JSON response is: “problem“. API applies several intelligent thread dump analysis patterns and if it detects any issues, it will reported in this “problem” element.

CURL command

Assuming your Thread dump file is located in “./my-thread-dump.txt,” then CURL command to invoke the API is:

curl -X POST –data-binary @./my-thread-dump.txt http://api.fastthread.io/fastthread-api?apiKey= –header “Content-Type:text”

It can’t get any more simpler than that? Isn’t it?

Other Tools

You can also invoke the API using any web service client tools such as: SOAP UI, Postman Browser Plugin,…..

postman-ft

Fig: POSTing Thread dumps through PostMan plugin

Sample Response

{
  "problem": [
    {
      "level": "SEVERE",
      "description": "8 thread are looping on same lines of code. If threads loop infinitely on the same lines of code, CPU consumption will start to spike up"
    }
  ],
  "threadsRemainingInWaitingState": [
    {
      "method": "java.lang.Object.wait(Native Method)",
      "threadCount": 3,
      "threads": "Reference Handler, Dispatcher-Thread-2, Finalizer"
    },
    {
      "method": "sun.misc.Unsafe.park(Native Method)",
      "threadCount": 2,
      "threads": "New Relic RPM Connection Service, New Relic Retransformer"
    }
  ],
  "threadDumpReport": [
    {
      "timestamp": "2016-03-03 10:37:28",
      "JVMType": " 64-Bit Server VM (23.7-b01 mixed mode)",
      "threadState": [
        {
          "state": "RUNNABLE",
          "threadCount": 28,
          "threads": "Attach Listener, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-B85-9, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-H87-3, InvoiceGeneratedQC-H87-1, InvoiceGeneratedQC-B85-9, Service Thread, C2 CompilerThread1, C2 CompilerThread0, Signal Dispatcher, main, VM Thread, GC task thread#0 (ParallelGC), GC task thread#1 (ParallelGC), GC task thread#2 (ParallelGC), GC task thread#3 (ParallelGC), GC task thread#4 (ParallelGC), GC task thread#5 (ParallelGC), GC task thread#6 (ParallelGC), GC task thread#7 (ParallelGC), GC task thread#8 (ParallelGC), GC task thread#9 (ParallelGC), GC task thread#10 (ParallelGC), GC task thread#11 (ParallelGC), GC task thread#12 (ParallelGC)"
        },
        {
          "state": "WAITING",
          "threadCount": 6,
          "threads": "Dispatcher-Thread-2, New Relic RPM Connection Service, New Relic Retransformer, Finalizer, Reference Handler, VM Periodic Task Thread"
        },
        {
          "state": "TIMED_WAITING",
          "threadCount": 4,
          "threads": "GC Daemon, New Relic Sampler Service, New Relic Deadlock Detector, New Relic Thread Service"
        }
      ],
      "repeatingStackTrace": [
        {
          "stackTrace": "stacktrace",
          "threadCount": 15,
          "threads": "VM Thread, GC task thread#0 (ParallelGC), GC task thread#1 (ParallelGC), GC task thread#2 (ParallelGC), GC task thread#3 (ParallelGC), GC task thread#4 (ParallelGC), GC task thread#5 (ParallelGC), GC task thread#6 (ParallelGC), GC task thread#7 (ParallelGC), GC task thread#8 (ParallelGC), GC task thread#9 (ParallelGC), GC task thread#10 (ParallelGC), GC task thread#11 (ParallelGC), GC task thread#12 (ParallelGC), VM Periodic Task Thread"
        },
        {
          "stackTrace": "java.lang.Thread.State: RUNNABLE
at com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.processTripType0(ItinerarySegmentProcessor.java:366)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.processItineraryByTripType(ItinerarySegmentProcessor.java:254)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.templateMethod(ItinerarySegmentProcessor.java:399)
...",
          "threadCount": 8,
          "threads": "InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-B85-9, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-H87-3, InvoiceGeneratedQC-H87-1, InvoiceGeneratedQC-B85-9"
        },
        {
          "stackTrace": "java.lang.Thread.State: RUNNABLE
",
          "threadCount": 5,
          "threads": "Attach Listener, Service Thread, C2 CompilerThread1, C2 CompilerThread0, Signal Dispatcher"
        },
        {
          "stackTrace": "java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for   (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
...",
          "threadCount": 3,
          "threads": "New Relic Sampler Service, New Relic Deadlock Detector, New Relic Thread Service"
        },
        {
          "stackTrace": "java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for   (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
...",
          "threadCount": 2,
          "threads": "New Relic RPM Connection Service, New Relic Retransformer"
        },
      ],
      "mostUsedMethod": [
        {
          "method": "com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380)",
          "threadCount": 8,
          "threads": "InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-B85-9, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-H87-3, InvoiceGeneratedQC-H87-1, InvoiceGeneratedQC-B85-9"
        },
        {
          "method": "sun.misc.Unsafe.park(Native Method)",
          "threadCount": 5,
          "threads": "New Relic RPM Connection Service, New Relic Sampler Service, New Relic Deadlock Detector, New Relic Thread Service, New Relic Retransformer"
        },
        {
          "method": "java.lang.Object.wait(Native Method)",
          "threadCount": 4,
          "threads": "Dispatcher-Thread-2, GC Daemon, Finalizer, Reference Handler"
        },
      ],
      "gcThreadsCount": 14
    },
    {
      "timestamp": "2016-03-03 10:37:38",
      "JVMType": " 64-Bit Server VM (23.7-b01 mixed mode)",
      "threadState": [
        {
          "state": "RUNNABLE",
          "threadCount": 28,
          "threads": "Attach Listener, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-B85-9, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-H87-3, InvoiceGeneratedQC-H87-1, InvoiceGeneratedQC-B85-9, Service Thread, C2 CompilerThread1, C2 CompilerThread0, Signal Dispatcher, main, VM Thread, GC task thread#0 (ParallelGC), GC task thread#1 (ParallelGC), GC task thread#2 (ParallelGC), GC task thread#3 (ParallelGC), GC task thread#4 (ParallelGC), GC task thread#5 (ParallelGC), GC task thread#6 (ParallelGC), GC task thread#7 (ParallelGC), GC task thread#8 (ParallelGC), GC task thread#9 (ParallelGC), GC task thread#10 (ParallelGC), GC task thread#11 (ParallelGC), GC task thread#12 (ParallelGC)"
        },
        {
          "state": "WAITING",
          "threadCount": 6,
          "threads": "Dispatcher-Thread-2, New Relic RPM Connection Service, New Relic Retransformer, Finalizer, Reference Handler, VM Periodic Task Thread"
        },
        {
          "state": "TIMED_WAITING",
          "threadCount": 4,
          "threads": "GC Daemon, New Relic Sampler Service, New Relic Deadlock Detector, New Relic Thread Service"
        }
      ],
      "repeatingStackTrace": [
        {
          "stackTrace": "stacktrace",
          "threadCount": 15,
          "threads": "VM Thread, GC task thread#0 (ParallelGC), GC task thread#1 (ParallelGC), GC task thread#2 (ParallelGC), GC task thread#3 (ParallelGC), GC task thread#4 (ParallelGC), GC task thread#5 (ParallelGC), GC task thread#6 (ParallelGC), GC task thread#7 (ParallelGC), GC task thread#8 (ParallelGC), GC task thread#9 (ParallelGC), GC task thread#10 (ParallelGC), GC task thread#11 (ParallelGC), GC task thread#12 (ParallelGC), VM Periodic Task Thread"
        },
        {
          "stackTrace": "java.lang.Thread.State: RUNNABLE
at com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.processTripType0(ItinerarySegmentProcessor.java:366)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.processItineraryByTripType(ItinerarySegmentProcessor.java:254)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.templateMethod(ItinerarySegmentProcessor.java:399)
...",
          "threadCount": 8,
          "threads": "InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-B85-9, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-H87-3, InvoiceGeneratedQC-H87-1, InvoiceGeneratedQC-B85-9"
        },
        {
          "stackTrace": "java.lang.Thread.State: RUNNABLE
",
          "threadCount": 5,
          "threads": "Attach Listener, Service Thread, C2 CompilerThread1, C2 CompilerThread0, Signal Dispatcher"
        },
        {
          "stackTrace": "java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for   (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
...",
          "threadCount": 3,
          "threads": "New Relic Sampler Service, New Relic Deadlock Detector, New Relic Thread Service"
        },
        {
          "stackTrace": "java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for   (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
...",
          "threadCount": 2,
          "threads": "New Relic RPM Connection Service, New Relic Retransformer"
        },
      ],
      "mostUsedMethod": [
        {
          "method": "com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380)",
          "threadCount": 8,
          "threads": "InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-B85-9, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-H87-3, InvoiceGeneratedQC-H87-1, InvoiceGeneratedQC-B85-9"
        },
        {
          "method": "sun.misc.Unsafe.park(Native Method)",
          "threadCount": 5,
          "threads": "New Relic RPM Connection Service, New Relic Sampler Service, New Relic Deadlock Detector, New Relic Thread Service, New Relic Retransformer"
        },
        {
          "method": "java.lang.Object.wait(Native Method)",
          "threadCount": 4,
          "threads": "Dispatcher-Thread-2, GC Daemon, Finalizer, Reference Handler"
        },
      ],
      "gcThreadsCount": 14
    }
  ],
  "responseId": "20161025113858_4"
}

JSON Response Elements

Element Data Type Description
responseId UUID Unique transaction Id that is generated for every response. This is used for any debugging or diagnosis purposes
fault If failed to process the request, then this element would be present
>reason Reason indicating why request couldn’t be processed
>details More details about the failure
graphURL Graphical visualization of the thread dumps can be found at this location.
problem Array This element is returned if any problems has been detected in the thread dumps. If more than one problem is detected then multiple ‘problem’ elements will be returned.
>level It can have value:

SEVERE

WARNING

>description Detailed description about the problem
threadsRemainingInRunnableState Array If more than one thread dump is sent in the request, this element will show the threads which are in RUNNABLE state across all the thread dumps. To understand why it’s important to track RUNNABLE threads across thread dumps, please check out Thread Mill thread dump analysis pattern
>threadCount Number of threads in RUNNABLE state
>threads Array Name of threads
>method Method in which threads remain in RUNNABLE state
threadsStuckInBlockedState Array If more than one thread dump is sent in the request, this element will show the threads which are in BLOCKED state across all the thread dumps. To understand why it’s important to track BLOCKED threads across thread dumps, please check out Atherosclerosis thread dump analysis pattern
>threadCount Number of threads in BLOCKED state
>threads Array Name of threads
>method Method in which threads remain in BLOCKED state
threadsRemainingInWaitingState Array If more than one thread dump is sent in the request, this element will show the threads which are in WAITING state across all the thread dumps.
>threadCount Number of threads in WAITING state
>threads Array Name of threads
>method Method in which threads remain in WAITING state
threadDumpReport Array If n number of thread dumps are sent in the request, then this element would be present n number of time. One ‘threadDumpReport’ element per thread dump sent in the request
>timestamp Timestamp at which thread dump was captured
>JVMType Information about the JVM from which thread dump was captured. JVM Version, client or server mode, 32 or 64-bit version
>problem Array This element is returned if any problems has been detected in this particular thread dump. If more than one problem is detected then multiple ‘problem’ elements will be returned.
>>level It can have value:

SEVERE

WARNING

>>description Detailed description about the problem
>threadState Array This element shows the threads grouped by the thread states.
>>threadCount Number of thread in a particular state
>>threads Array Name of the threads in this particular state
>>state It can have one of the following value:

NEW,

RUNNABLE,

BLOCKED,

WAITING,

TIMED_WAITING,

TERMINATED,

IN_NATIVE,

UNABLE_DETERMINE

>repeatingStackTrace Array Threads that have same stack trace are reported here. To understand it’s important to study the threads with same stack trace refer to RSI thread dump analysis pattern
>>threadCount Number of threads in this particular stack trace
>>threads Array Name of the threads in this particular state
>>stackTrace Stack Trace of the threads
>mostUsedMethod Threads that are in the same method are reported here. To understand it’s important to study the threads in same method refer to All Roads Lead to Rome thread dump analysis pattern
>>threadCount Number of threads in this particular method
>>threads Array Name of the threads in this particular method
>>method Method Name
>blockedBlocker Array This element shows the threads that transitively stuck by a lock. To learn more please refer to Traffic Jam thread dump analysis pattern.
>>blockedThreadCount Number of threads that transitively stuck by this lock
>>lock Lock in which threads are stuck
>gcThreadsCount

Total number of Garbage Collection threads in the JVM. Too many GC threads can hurt JVM’s performance. To learn more check Several Scavengers thread dump analysis pattern

Advertisements