Fast thread

In this modern world, thread dumps are still analyzed in a tedious & manual mode i.e., you have to get hold of DevOps team, ask them to send you the

thread dumps, then they will mail you the thread dumps, then you will upload the dumps in to a thread dump analysis tool, then you have to apply your intelligence to analyze it. There is no programmatic way to analyze thread dumps in a proactive manner. Thus to eliminate this hassle, fastthread.io is introducing a RESTful API to analyze thread dumps. With one line of CURL command, you can get your thread dumps analyzed instantly.

Here are a few use cases where this API can be extremely useful.

Use case 1: Automatic Root cause Analysis

Most of the DevOps invokes a simple Http ping or APM tools to monitor the applications health. This ping is good to detect whether application is alive or not. APM tools are great at informing that application’s CPU spiked  up by ‘x%’, memory utilization increased by ‘y%’, response time dropped by ‘z’ milliseconds. It won’t inform what caused the CPU to spike up, what caused memory utilization to increase, what caused the response time to degrade. If you can configure Cron job to capture thread dumps/GC logs on a periodic interval and invoke our REST API, we apply our intelligent patterns & machine learning algorithms to instantly identify the root cause of the problem.

Advantage 1: Whenever these sort of production problem happens, because of the heat of the moment, DevOps team recycles the servers with out capturing the thread dumps and GC logs. You need to capture thread dumps and GC logs at the moment when problem is happening, in order to diagnose the problem. In this new strategy you don’t have to worry about it, because your cron job is capturing thread dumps/GC logs on a periodic intervals and invoking the REST API, all your thread dumps/GC Logs are archived in our servers.

Advantage 2: Unlike APM tools which claims to add less than 3% of overhead, where as in reality it adds multiple folds, beauty of this strategy is: It doesn’t add any overhead (or negligible overhead). Because entire analysis of the thread dumps/GCeasy are done on our servers and not on your production servers..

Use case 2: Performance Tests

When you conduct performance tests, you might want to take thread dumps/GC logs on a periodic basis and get it analyzed through the API. In case if thread count goes beyond a threshold or if too many threads are WAITING or if any threads are BLOCKED for a prolonged period or lock isn’t getting released or frequent full GC activities happening or GC pause time exceeds thresholds, it needs to get the visibility right then and there. It should be analyzed before code hits the production. In such circumstance this API will become very handy.

Use case 3: Continuous Integration

As part of continuous integration it’s highly encouraged to execute performance tests. Thread dumps/GC Logs should be captured and it can be analyzed using the API.  If API reports any problems, then build can be failed. In this way, you can catch the performance degradation right during code commit time instead of catching it in performance labs or production.

How to invoke Thread dump analysis API?

Invoking Thread dump analysis API is very extremely simple:

1. Once you Sign in to fastThread, click on the ‘API’ tab. There you will find the REST API URL endpoint. 
2. Post HTTP request to https://api.fastthread.io/fastthread-api?apiKey={YOUR_API_KEY}
   Your API key will be mentioned in your fastThread dashboard under the ‘API’ tab.
3. The body of the HTTP request should contain the Thread dump that needs to be analyzed. You can either send 1 thread dump or multiple thread dumps in the same request.
4. HTTP Response will be sent back in JSON format. JSON has several important stats about the Thread dump. Primary element to look in the JSON response is: “problem“. API applies several intelligent thread dump analysis patterns and if it detects any issues, it will reported in this “problem” element.

CURL command

Assuming your Thread dump file is located in “./my-thread-dump.txt,” then CURL command to invoke the API is:

curl -X POST --data-binary @./my-thread-dump.txt https://api.fastthread.io/fastthread-api?apiKey={YOUR_API_KEY} --header "Content-Type:text"

It can’t get any more simpler than that? Isn’t it?

Compression

Thread Dump are quite large in size. For fast and efficient processing, we recommend you to compress and send the Thread dump files. When you are compressing the Thread dump, you need to pass ‘Content-Encoding’ element in the HTTP Header element or in the URL parameter.

Say suppose you are compressing Thread dump file in to ‘zip’ format, then you can invoke the API with HTTP header element

curl -X POST --data-binary @./my-thread-dump.zip "https://api.fastthread.io/fastthread-api?apiKey={YOU_API_KEY}" --header "Content-Type:text" --header "Content-Encoding:zip"

or you can also invoke the API with ‘Content-Encoding’ element in the URL parameter

curl -X POST --data-binary @./my-thread-dump.zip "https://api.fastthread.io/fastthread-api?apiKey={YOUR_API_KEY}&Content-Encoding=zip" --header "Content-Type:text"

We support following compression formats:

zip, gz, xz, z, lzma, deflate, sz, lz4, zstd, bz2, tar

You may use the one of your choice. Whatever compression format you used for compressing the thread dump should be passed in ‘Content-Encoding’ element.

Downloading Thread Dump from remote location

Say suppose, you have stored your thread dump files in a remote location like AWS S3 bucket. In those cases, if you want to analyze your thread dump then you will have to download files to your local machine and then invoke the API with thread dump file. To make it simple and automation easier we have introduced ‘location’ parameter in our API. You can pass http(s) URL location to the ‘location’ parameter and invoke the API. When you do it, API will download file from the remote location, analyze the data and return back the response. For more details refer to this article.

curl -X POST "https://api.fastthread.io/fastthread-api?apiKey={YOU_API_KEY}&location={THREAD-DUMP-HTTP-URL}" --header "Content-Type:text"

Other Tools

You can also invoke the API using any web service client tools such as: SOAP UI, Postman Browser Plugin,…..

Fig: POSTing Thread dumps through PostMan plugin

Sample Response

{
  "problem": [
    {
      "level": "SEVERE",
      "description": "8 thread are looping on same lines of code. If threads loop infinitely on the same lines of code, CPU consumption will start to spike up"
    }
  ],
  "threadsRemainingInWaitingState": [
    {
      "method": "java.lang.Object.wait(Native Method)",
      "threadCount": 3,
      "threads": "Reference Handler, Dispatcher-Thread-2, Finalizer"
    },
    {
      "method": "sun.misc.Unsafe.park(Native Method)",
      "threadCount": 2,
      "threads": "New Relic RPM Connection Service, New Relic Retransformer"
    }
  ],
  "threadDumpReport": [
    {
      "timestamp": "2016-03-03 10:37:28",
      "JVMType": " 64-Bit Server VM (23.7-b01 mixed mode)",
      "threadState": [
        {
          "state": "RUNNABLE",
          "threadCount": 28,
          "threads": "Attach Listener, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-B85-9, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-H87-3, InvoiceGeneratedQC-H87-1, InvoiceGeneratedQC-B85-9, Service Thread, C2 CompilerThread1, C2 CompilerThread0, Signal Dispatcher, main, VM Thread, GC task thread#0 (ParallelGC), GC task thread#1 (ParallelGC), GC task thread#2 (ParallelGC), GC task thread#3 (ParallelGC), GC task thread#4 (ParallelGC), GC task thread#5 (ParallelGC), GC task thread#6 (ParallelGC), GC task thread#7 (ParallelGC), GC task thread#8 (ParallelGC), GC task thread#9 (ParallelGC), GC task thread#10 (ParallelGC), GC task thread#11 (ParallelGC), GC task thread#12 (ParallelGC)"
        },
        {
          "state": "WAITING",
          "threadCount": 6,
          "threads": "Dispatcher-Thread-2, New Relic RPM Connection Service, New Relic Retransformer, Finalizer, Reference Handler, VM Periodic Task Thread"
        },
        {
          "state": "TIMED_WAITING",
          "threadCount": 4,
          "threads": "GC Daemon, New Relic Sampler Service, New Relic Deadlock Detector, New Relic Thread Service"
        }
      ],
      "repeatingStackTrace": [
        {
          "stackTrace": "stacktrace",
          "threadCount": 15,
          "threads": "VM Thread, GC task thread#0 (ParallelGC), GC task thread#1 (ParallelGC), GC task thread#2 (ParallelGC), GC task thread#3 (ParallelGC), GC task thread#4 (ParallelGC), GC task thread#5 (ParallelGC), GC task thread#6 (ParallelGC), GC task thread#7 (ParallelGC), GC task thread#8 (ParallelGC), GC task thread#9 (ParallelGC), GC task thread#10 (ParallelGC), GC task thread#11 (ParallelGC), GC task thread#12 (ParallelGC), VM Periodic Task Thread"
        },
        {
          "stackTrace": "java.lang.Thread.State: RUNNABLE
at com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.processTripType0(ItinerarySegmentProcessor.java:366)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.processItineraryByTripType(ItinerarySegmentProcessor.java:254)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.templateMethod(ItinerarySegmentProcessor.java:399)
...",
          "threadCount": 8,
          "threads": "InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-B85-9, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-H87-3, InvoiceGeneratedQC-H87-1, InvoiceGeneratedQC-B85-9"
        },
        {
          "stackTrace": "java.lang.Thread.State: RUNNABLE
",
          "threadCount": 5,
          "threads": "Attach Listener, Service Thread, C2 CompilerThread1, C2 CompilerThread0, Signal Dispatcher"
        },
        {
          "stackTrace": "java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for   (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
...",
          "threadCount": 3,
          "threads": "New Relic Sampler Service, New Relic Deadlock Detector, New Relic Thread Service"
        },
        {
          "stackTrace": "java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for   (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
...",
          "threadCount": 2,
          "threads": "New Relic RPM Connection Service, New Relic Retransformer"
        },
      ],
      "mostUsedMethod": [
        {
          "method": "com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380)",
          "threadCount": 8,
          "threads": "InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-B85-9, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-H87-3, InvoiceGeneratedQC-H87-1, InvoiceGeneratedQC-B85-9"
        },
        {
          "method": "sun.misc.Unsafe.park(Native Method)",
          "threadCount": 5,
          "threads": "New Relic RPM Connection Service, New Relic Sampler Service, New Relic Deadlock Detector, New Relic Thread Service, New Relic Retransformer"
        },
        {
          "method": "java.lang.Object.wait(Native Method)",
          "threadCount": 4,
          "threads": "Dispatcher-Thread-2, GC Daemon, Finalizer, Reference Handler"
        },
      ],
            "threadGroup": [
                {
                    "group": "DQBFacade",
                    "threadCount": 100
                },
                {
                    "group": "SectorwiseContractsFacade",
                    "threadCount": 100
                },
                {
                    "group": "SameDayVoidQC",
                    "threadCount": 100
                },
                {
                    "group": "GC task thread",
                    "threadCount": 13
                },
                {
                    "group": "DefaultQuartzScheduler_Worker",
                    "threadCount": 10
                },
            ],
      "gcThreadsCount": 14
    },
    {
      "timestamp": "2016-03-03 10:37:38",
      "JVMType": " 64-Bit Server VM (23.7-b01 mixed mode)",
      "threadState": [
        {
          "state": "RUNNABLE",
          "threadCount": 28,
          "threads": "Attach Listener, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-B85-9, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-H87-3, InvoiceGeneratedQC-H87-1, InvoiceGeneratedQC-B85-9, Service Thread, C2 CompilerThread1, C2 CompilerThread0, Signal Dispatcher, main, VM Thread, GC task thread#0 (ParallelGC), GC task thread#1 (ParallelGC), GC task thread#2 (ParallelGC), GC task thread#3 (ParallelGC), GC task thread#4 (ParallelGC), GC task thread#5 (ParallelGC), GC task thread#6 (ParallelGC), GC task thread#7 (ParallelGC), GC task thread#8 (ParallelGC), GC task thread#9 (ParallelGC), GC task thread#10 (ParallelGC), GC task thread#11 (ParallelGC), GC task thread#12 (ParallelGC)"
        },
        {
          "state": "WAITING",
          "threadCount": 6,
          "threads": "Dispatcher-Thread-2, New Relic RPM Connection Service, New Relic Retransformer, Finalizer, Reference Handler, VM Periodic Task Thread"
        },
        {
          "state": "TIMED_WAITING",
          "threadCount": 4,
          "threads": "GC Daemon, New Relic Sampler Service, New Relic Deadlock Detector, New Relic Thread Service"
        }
      ],
      "repeatingStackTrace": [
        {
          "stackTrace": "stacktrace",
          "threadCount": 15,
          "threads": "VM Thread, GC task thread#0 (ParallelGC), GC task thread#1 (ParallelGC), GC task thread#2 (ParallelGC), GC task thread#3 (ParallelGC), GC task thread#4 (ParallelGC), GC task thread#5 (ParallelGC), GC task thread#6 (ParallelGC), GC task thread#7 (ParallelGC), GC task thread#8 (ParallelGC), GC task thread#9 (ParallelGC), GC task thread#10 (ParallelGC), GC task thread#11 (ParallelGC), GC task thread#12 (ParallelGC), VM Periodic Task Thread"
        },
        {
          "stackTrace": "java.lang.Thread.State: RUNNABLE
at com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.processTripType0(ItinerarySegmentProcessor.java:366)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.processItineraryByTripType(ItinerarySegmentProcessor.java:254)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.templateMethod(ItinerarySegmentProcessor.java:399)
...",
          "threadCount": 8,
          "threads": "InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-B85-9, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-H87-3, InvoiceGeneratedQC-H87-1, InvoiceGeneratedQC-B85-9"
        },
        {
          "stackTrace": "java.lang.Thread.State: RUNNABLE
",
          "threadCount": 5,
          "threads": "Attach Listener, Service Thread, C2 CompilerThread1, C2 CompilerThread0, Signal Dispatcher"
        },
        {
          "stackTrace": "java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for   (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
...",
          "threadCount": 3,
          "threads": "New Relic Sampler Service, New Relic Deadlock Detector, New Relic Thread Service"
        },
        {
          "stackTrace": "java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for   (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
...",
          "threadCount": 2,
          "threads": "New Relic RPM Connection Service, New Relic Retransformer"
        },
      ],
      "mostUsedMethod": [
        {
          "method": "com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380)",
          "threadCount": 8,
          "threads": "InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-B85-9, InvoiceGeneratedQC-A99-6, InvoiceGeneratedQC-H87-6, InvoiceGeneratedQC-H87-3, InvoiceGeneratedQC-H87-1, InvoiceGeneratedQC-B85-9"
        },
        {
          "method": "sun.misc.Unsafe.park(Native Method)",
          "threadCount": 5,
          "threads": "New Relic RPM Connection Service, New Relic Sampler Service, New Relic Deadlock Detector, New Relic Thread Service, New Relic Retransformer"
        },
        {
          "method": "java.lang.Object.wait(Native Method)",
          "threadCount": 4,
          "threads": "Dispatcher-Thread-2, GC Daemon, Finalizer, Reference Handler"
        },
      ],
      "gcThreadsCount": 14
    }
  ],
  "responseId": "20161025113858_4"
}

JSON Response Elements

Element	Data Type	Description
responseId	UUID	Unique transaction Id that is generated for every response. This is used for any debugging or diagnosis purposes
fault		If failed to process the request, then this element would be present
>reason		Reason indicating why request couldn’t be processed
>details		More details about the failure
graphURL		Graphical visualization of the thread dumps can be found at this location.
problem	Array	This element is returned if any problems has been detected in the thread dumps. If more than one problem is detected then multiple ‘problem’ elements will be returned.
>level		It can have one of the following value based on the severity of the problem: FATAL SEVERE WARNING
>description		Detailed description about the problem
threadsRemainingInRunnableState	Array	If more than one thread dump is sent in the request, this element will show the threads which are in RUNNABLE state across all the thread dumps. To understand why it’s important to track RUNNABLE threads across thread dumps, please check out Thread Mill thread dump analysis pattern
>threadCount		Number of threads in RUNNABLE state
>threads	Array	Name of threads
>method		Method in which threads remain in RUNNABLE state
threadsStuckInBlockedState	Array	If more than one thread dump is sent in the request, this element will show the threads which are in BLOCKED state across all the thread dumps. To understand why it’s important to track BLOCKED threads across thread dumps, please check out Atherosclerosis thread dump analysis pattern
>threadCount		Number of threads in BLOCKED state
>threads	Array	Name of threads
>method		Method in which threads remain in BLOCKED state
threadsRemainingInWaitingState	Array	If more than one thread dump is sent in the request, this element will show the threads which are in WAITING state across all the thread dumps.
>threadCount		Number of threads in WAITING state
>threads	Array	Name of threads
>method		Method in which threads remain in WAITING state
threadDumpReport	Array	If n number of thread dumps are sent in the request, then this element would be present n number of time. One ‘threadDumpReport’ element per thread dump sent in the request
>timestamp		Timestamp at which thread dump was captured
>JVMType		Information about the JVM from which thread dump was captured. JVM Version, client or server mode, 32 or 64-bit version
>threadCount	Integer	Total number of threads in this thread dump
>problem	Array	This element is returned if any problems has been detected in this particular thread dump. If more than one problem is detected then multiple ‘problem’ elements will be returned.
>>level		It can have one of the following value based on the severity of the problem: FATAL SEVERE WARNING
>>description		Detailed description about the problem
>threadState	Array	This element shows the threads grouped by the thread states.
>>threadCount		Number of thread in a particular state
>>threads	Array	Name of the threads in this particular state
>>state		It can have one of the following value: NEW, RUNNABLE, BLOCKED, WAITING, TIMED_WAITING, TERMINATED, IN_NATIVE, UNABLE_DETERMINE
>repeatingStackTrace	Array	Threads that have same stack trace are reported here. To understand it’s important to study the threads with same stack trace refer to RSI thread dump analysis pattern
>>threadCount		Number of threads in this particular stack trace
>>threads	Array	Name of the threads in this particular state
>threadGroup	Array	This element shows the thread groups that are present in teh application
>>threadCount		Number of thread in a particular thread group
>>group		Name of the thread group
>repeatingStackTrace	Array	Threads that have same stack trace are reported here. To understand it’s important to study the threads with same stack trace refer to RSI thread dump analysis pattern
>>threadCount		Number of threads in this particular stack trace
>>threads	Array	Name of the threads in this particular state
>>stackTrace		Stack Trace of the threads
>mostUsedMethod		Threads that are in the same method are reported here. To understand it’s important to study the threads in same method refer to All Roads Lead to Rome thread dump analysis pattern
>>threadCount		Number of threads in this particular method
>>threads	Array	Name of the threads in this particular method
>>method		Method Name
>blockedBlocker	Array	This element shows the threads that transitively stuck by a lock. To learn more please refer to Traffic Jam thread dump analysis pattern.
>>blockedThreadCount		Number of threads that transitively stuck by this lock
>>lock		Lock in which threads are stuck
>gcThreadsCount		Total number of Garbage Collection threads in the JVM. Too many GC threads can hurt JVM’s performance. To learn more check Several Scavengers thread dump analysis pattern

Tagging the report

Suppose you are analyzing thread dumps across hundreds of JVM. You want to tag each report with the hostname, date,… so that it will be handy to know which report belongs to which host. This can be achieved by passing request parameter ‘fileName‘ in the API endpoint i.e. 

https://api.fastthread.io/fastthread-api?apiKey={YOUR_API_KEY}&fileName={YOUR_TAG}

Response JSON contains ‘webReport’ element. This element contains an URL. When you paste this URL in the browser, you will see the visual report of the thread dump that was parsed through API. Now on the right top corner of the report in the browser, you will be able to see the tag you had passed in the API.

Say suppose if you pass fileName=Production_Env_Server_1 in the API endpoint, this is how it will show up in the report.

Fig: FastThread report with the tag passed in API

XML Response

By default API response in sent in JSON format. However if you would like to get response in XML format, you can pass ‘accept’ HTTP header element with value ‘xml’

Curl –X POST --data-binary @./my-thread-dump.hprof https://api.fastthread.io/fastthread-api?apiKey={API_KEY_SENT_IN_EMAIL} –-header "accept:xml"

or you can also invoke the API with ‘accept’ element in the URL parameter

Curl –X POST --data-binary @./my-thread-dump.hprof https://api.fastthread.io/fastthread-api?apiKey={YOUR_API_KEY}&accept=xml

Key elements in API Response

FastThread’s JSON APIs are used for application monitoring, analyzing code quality during CI/CD pipeline and several other purposes. API response contains rich set of information (i.e. lot of elements). In this article, we have highlighted few key elements in the API response. If values of these elements exceed or drops below certain threshold, you might consider raising alerts/warnings or breaking the build.