I told my manager that I like the title of this article. He said to me that’s not surprising, because I am still an eligible bachelor. No wonder I like the word “marrying” in the title. But for folks like him who are married for 16 years, this title may give nightmares :-). Thus half-heartedly, I have chosen this title and continue to write on it. 

In this blog, I would like to show how to merge thread dump and top -H output and get meaningful results. These meaningful results will be highly helpful to troubleshoot CPU spike problems. Here is a real life case study, where a major trading application in North America benefited from this strategy. (We recommend you to read this case study, before continuing to read this article).

Step 1: Capturing Thread dumps

The first step is to capture the thread dump from your application, which is suffering from CPU spike. There are 8 different ways to capture the thread dump. You can use the option which is convenient for you.

Step 2: Capture ‘top -H’ output

Once thread dumps are captured, the next step is to capture the top -H output. When the ‘top’ tool is invoked with this ‘-H’ option, it will display all the threads that are running within that particular process. It will also display the amount of CPU and memory consumed by each thread within that process.

Below is the command you need to execute to capture top -H output.

$ top -H -p {PID} > my-top-H.txt

where “{PID}” is your application’s process ID. 

where “my-top-H.txt” is the file where top -H output will be stored.

Step 3: Analyzing thread dumps and Top -H output

Now we have captured thread dump and top -H output. The next step is to analyze this data. For this purpose, we will use the FastThread tool. 

  1. Open the FastThread application in your browser. 
  2. Select Thread dump(s) that you captured in step #1.
  3. Click on Add File button.
  4. Now select top -H output file that you captured in step #2.
  5. Click on the Analyze button.

Fig: Upload box in FastThread home page

The tool will analyze both files and generate a beautiful report like this. The report contains ‘CPU | Memory’ section. This is the section where the tool marries top -H output and thread dump and provides a CPU & memory consumed by each thread in the application. From this section, you will be able to see the top CPU consuming threads and the exact lines of code they are executing.

Fig: CPU, Memory consumed by each thread (generated by fastThread)

REST API

Instead of manually collecting the top -H output, thread dumps and then uploading to FastThread tool to see the result, you can analyze in an automated manner as well using the thread dump analysis API. 

Here are the steps to analyze the data through REST API:

  1. Create a new file 
  2. In this file at the top part paste your thread dump(s) data.
  3. After that append top -H output (below thread dump data).
  4. Invoke the FastThread REST API as outlined here, using this newly created file.

API will analyze the thread dumps, ‘top -H’ output and send back the metrics in the API response. This is the same metrics that you will see in the browser when you upload the dumps manually.