It’s late at night and a critical system has hung. What tools do you have on hand to detect the problem? And what utilities are worth including in your toolbox beforehand, so you’re prepared for this type of problem?
Thread dump analyzers are, of course, the solution, but if you don’t have one, all is not lost: the JDK provides some excellent tools for detecting deadlocks and hung threads.
This article looks at 5 problem-solving applications you probably already have, or, if you don’t, they’re available either in the cloud or as a free download.
What Causes Deadlocks and Program Hangs?
A deadlock occurs when, for example:
- Thread A holds a lock to Lock 1 AND
- Thread A requests a lock to Lock 2 AND
- Thread B holds a lock to Lock 2 AND
- Thread B requests a lock to Lock 1.
Neither thread will ever be able to move forward, and they will both hang until the JVM is killed.
A hung thread occurs when the thread is waiting for a resource that’s not currently available. For example, it may be waiting for a lock, a resource such as an I/O device, or response from a request sent to an external destination. If the resource becomes available, the thread will continue; if not, it will hang forever. A common example of this is where there is too much congestion on database server requests.
The program may also appear to hang if one thread is consuming too much CPU time, blocking other threads frequently.
Sample Programs
In this article, we’ll use two sample programs to try out 5 different thread analysis tools.
Program 1 deliberately creates a deadlock. It creates two lock objects, then starts two threads. The first, threadA, claims the first lock, then attempts to claim the second. The other, threadB, does the same thing, but in reverse order. Neither thread will be able to progress, because they are deadlocked.
// Create 2 threads that will deadlock
//====================================
public class BuggyProg12 {
Object lockA = new Object();
Object lockB = new Object();
public static void main(String[] args) {
BuggyProg12 b = new BuggyProg12();
}
public BuggyProg12() {
Thread threadA = new Thread(() -> {
synchronized (lockA) {
try{Thread.sleep(10);} catch(Exception e){}
synchronized (lockB) {
try{Thread.sleep(10);} catch(Exception e){}
}
}
});
Thread threadB = new Thread(() -> {
synchronized (lockB) {
try{Thread.sleep(10);} catch(Exception e){}
synchronized (lockA) {
try{Thread.sleep(10);} catch(Exception e){}
}
}
});
threadA.start();
threadB.start();
}
}Program 2 simulates a blocked thread, as well as a CPU-hungry thread. It first places a lock on an object named lock. It then creates a thread, threadA, which repeatedly carries out a calculation in a tight loop. A second thread, threadB, tries to claim lock, but will be blocked. Finally, the program waits for several minutes to allow time for diagnostics to be run.
// Create 1 thread that blocks and 1 that uses excessive CPU
//==========================================================
public class BuggyProg13 {
static Object lock1 = new Object();
public static void main(String[] args) {
synchronized(lock1) {
BuggyProg13 b = new BuggyProg13();
// Sleep for long enough for diagnostics to be taken
try{Thread.sleep(500000);} catch(Exception e){}
}
}
public BuggyProg13() {
Thread threadA = new Thread(() -> {
long a=0;
for (long i=0;i<Long.MAX_VALUE;i++)
a = i-5;
});
Thread threadB = new Thread(() -> {
System.out.println("Attemting to synchronize");
synchronized(lock1) {
System.out.println("Synchronized");
}
});
threadA.start();
threadB.start();
}
}Detecting Deadlocks and Hung Threads: CLI tools, Visual Monitors and Thread Dump Analyzers
A wide range of tools exist to help with diagnosing these issues. From simple command-line utilities available with the JDK to full-featured thread dump analyzers such as fastThread, these tools can detect deadlocks instantly, and assist with other aspects of thread health
Our choice of tool may depend on what utilities are immediately on hand in a crisis. The type of JDK we’re using is also a factor, as is personal preference for familiar tools.
Let’s take a brief look at some of the choices.
1. JStack
This utility is part of the JDK, so it should always be readily available.
JStack creates a thread dump in readable text format, which you can either examine manually, or submit to a thread dump analyzer. For large programs, it may be thousands of lines long, so it’s time-consuming to use it for deep analysis. On the other hand, it highlights deadlocks at the end of the dump, so it’s probably the fastest and easiest way to check whether an application is in this state.
The thread dump contains information about each thread including:
- Meta data;
- Thread state (e.g. running, blocked, waiting);
- Locks each thread is waiting for;
- Locks each thread holds;
- Stack Traces;
- CPU time used.
Additionally, if there is a deadlock, the report will end with full details. If we run Program 1 and use jstack to take a thread dump, the last part of the output looks like this:
Found one Java-level deadlock:
=============================
"Thread-0":
waiting to lock monitor 0x00000223d11a3100 (object 0x0000000089c0d5f0, a java.lang.Object),
which is held by "Thread-1"
"Thread-1":
waiting to lock monitor 0x00000223d0762bb0 (object 0x0000000089c0d5e0, a java.lang.Object),
which is held by "Thread-0"
Java stack information for the threads listed above:
===================================================
"Thread-0":
at BuggyProg12.lambda$new$0(BuggyProg12.java:17)
- waiting to lock <0x0000000089c0d5f0> (a java.lang.Object)
- locked <0x0000000089c0d5e0> (a java.lang.Object)
at BuggyProg12$$Lambda$1/0x0000000800c00a08.run(Unknown Source)
at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)
"Thread-1":
at BuggyProg12.lambda$new$1(BuggyProg12.java:26)
- waiting to lock <0x0000000089c0d5e0> (a java.lang.Object)
- locked <0x0000000089c0d5f0> (a java.lang.Object)
at BuggyProg12$$Lambda$2/0x0000000800c00c28.run(Unknown Source)
at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)
Found 1 deadlock.This gives us everything we need to resolve a deadlock.
Hung threads and threads using too much CPU require more effort, though it’s still possible to identify them by checking whether they are blocked or waiting, and comparing the amount of CPU they use.
To obtain a thread dump to a file, the command is:
jstack <pid> > <filename>We can also obtain the same thread dump by using the command line utility jcmd, which is also available with the JDK. The command is:
jcmd <pid> Thread.print > <filename>2. JConsole
This is a GUI-based utility that is also packaged with the JDK. To run it, type jconsole from the command line. It will then open a GUI window, which allows us to select a running Java application.
We can use it to monitor various aspects of the JVM.
First let’s see how it detects the deadlock in Program 1. If we click the Threads tab, it has a button at the bottom labelled ‘Detect Deadlock’. Clicking this displays the screen shown below. It lists the deadlocked threads. We can click on a thread to see why it’s blocked, and who’s holding the lock. It also displays the stack trace.
Fig: JConsole Showing Deadlocked Threads.
It’s less useful for finding blocked threads and threads with high CPU usage, but we can click on any thread to see its status and stack trace. The screenshot below shows the details for the blocked thread thread1 in Program 2.
Fig: JConsole Showing a Blocked Thread
Incidentally, it’s helpful in a large program to make sure all threads and inner classes are named to make debugging easier.
3. VisualVM
VisualVM was originally distributed with the JDK as JVisualVM, but later versions don’t include it. It’s now available from VisualVM’s GitHub page.
It’s a GUI-based executable program. Like JConsole, it allows us to select a running JVM, and has several monitoring features.
VisualVM instantly detects and reports deadlocks, and has the facility to take and view a thread dump to explore the issue further. See the screenshot below taken when monitoring Program 1.
Fig: Deadlock Detected by JVisualVM
When monitoring Program 2, which has one blocked thread and one using excess CPU, the monitoring screen looks like this:
Fig: JVisualVM Summary for Program 2
The blocked thread is shown in blue, labelled ‘Monitor.’ Other than the facility to take a thread dump, it doesn’t give the opportunity to explore further, or identify high CPU usage.
4. IBM TMDA (Thread Monitor and Dump Analyzer)
This is specifically designed for applications running IBM’s JVM, but does have limited functionality for other Java platforms such as HotSpot.
It’s a thread dump analyzer, so we would first need to use a tool such as JStack, Jcmd or VisualVM to dump running threads.
TMDA runs via a downloadable .jar file, and has a GUI screen that allows us to upload thread dumps.
When we upload the dump from Program 1, it instantly detects that there is a deadlock, and identifies the threads concerned.
Fig: IBM TMDA Showing Deadlocked Threads
It allows us to explore individual threads to view their status, what locks they own and what locks they are waiting for. It also shows a stack trace, as shown below.
Fig: Thread Explorer in IBM TMDA
We can also use this explorer to browse through the threads in Program 2, and identify what Thread-1 is waiting for.
Fig: IBM TMDA Thread Explorer
When working on dumps taken from non-IBM JVMs, it’s not able to analyze CPU usage.
5. fastThread Thread Dump Analyzer
This is a sophisticated tool that can carry out deep analysis of thread behavior. It has a free version, which allows us to upload thread dumps into the cloud for analysis, and a facility to set up limited automatic monitoring via REST APIs. The paid version is able to diagnose many issues via machine learning algorithms, has more comprehensive APIs, and also has the option of an on-premise version.
When the dump from Program 1 is uploaded to the fastThread site, a message appears at the front of the report as shown below:
Fig: fastThread Reports Deadlock
We can click on the link to view details of the threads concerned, as shown below:
Fig: Details of Deadlocked Threads Displayed by fastThread
We can then use the stack traces to find the rogue code.
When we upload the dump from Program 2, we first see a warning indicating high CPU usage, as shown below.
Fig: fastThread Indicates High CPU Consumption
We can then explore the CPU Consuming Threads section of the report to view details.
Fig: CPU-Consuming Threads
To investigate the second problem in this program, which is a blocked thread, we first look at the thread summary, which shows one thread as blocked.
Fig: fastThread Summary
We can click on the button to view its details, as shown below:
Fig: Details of Blocked Thread
For a demonstration of fastThread in action, you may like this video: How to Use fastThread.
Conclusion
All the tools we looked at are able to identify deadlocks instantly, saving hours of searching through code when a program hangs. The tools provided by the JDK, which include JStack and JConsole, are available on any JDK installation. VisualVM and IBM TMDA must be downloaded and installed before we can use them.
We can also access fastThread instantly in the cloud, making it available wherever a problem occurs. It has the additional advantage that it can identify other thread-related issues very quickly. With this tool we were able to diagnose blocked and high CPU-using threads within minutes.
Share your Thoughts!