Fast thread

Analyzing thread activity is an important part of troubleshooting Java applications. It’s simple to extract a thread dump from a running program, and, since the dump file is in text format, it’s feasible to interpret it manually for a small program. For complex programs, the thread dump may be thousands of lines long. Java thread dump analyzer tools have evolved to take the pain out of extracting meaningful information from dumps.

Which thread dump analyzer is best for your organization? Let’s compare some of the top tools, both free and paid, that are currently available.

When Would We Need to Use a Thread Dump Analyzer?

Even if a program doesn’t explicitly use multithreading, it will still have many threads running. The JVM uses background tasks for operations such as garbage collection and pre-compiling hot code.

Modern applications are often event-driven, and will generally process requests in a separate thread, leaving the main program free to deal with incoming events. This is true of a wide range of systems, including web servers, GUI-based programs, robotics and process control applications.

When we experience issues such as slow response, program hangs and high CPU usage, thread dump analyzers like fastThread can often help us to find the problem quickly. They’re also useful for investigating thread-related errors, such as java.lang.OutOfMemoryError: Unable to create new native threads.

To learn how to take a thread dump, read this article: 7 Options for Taking a Thread Dump. For information about what a thread dump contains, see Understanding Java Thread Dumps.

How Can We Use a Thread Dump Analyzer to Troubleshoot Java Issues?

Unless the program has actually crashed, it’s best to take at least three thread dumps when the application issue is at its worst. They can be taken quite close together – perhaps every ten seconds. This enables us to see which threads are progressing and which are not.

We can use dump analyzers in several ways, depending on the issue. The first step is to try to identify the thread that’s causing the problem. Once that’s established, the dump contains a stack trace for each thread, and it’s a simple matter to follow the program path to see where it’s going wrong.

If we’re investigating slow performance, a few things to look at are:

• What were the last-executed methods of each thread? Were several threads executing the same method? If so, it’s likely to be the cause of the bottleneck. This often happens if there’s contention for resources such as databases or network connections. A flame graph can be very helpful in identifying suspect code.
• Investigate blocked or waiting threads. A good tool should make it easy to identify what’s blocking each thread.

When CPU usage is high, we need to identify which threads are using the most CPU. In Linux, we can easily do this using the command top -H <pid>. From there, we can tie the thread IDs back to the actual threads in the dump. A good analyzer tool also identifies threads that are using a lot of CPU time. Again, once we’ve identified the thread, the stack trace will let us follow the code path taken to reach the current line of code.

Garbage collection issues can also cause high CPU usage. From the thread dump, we can see the GC threads. Are there too many? This can often happen in containers. Too few? It’s also worth analyzing GC logs using a tool such as GCeasy.

If the system is hanging, we need to check for deadlocks, where, for example, Thread A holds a lock that’s needed by Thread B and vice versa. When this happens, neither can progress, and we may have to bring down the JVM. A good tool will identify deadlocks, if there are any. Program hangs can also be caused by too many blocked threads, or high contention for CPU time.

Other things to check for are:

• Threads with a very long stack trace. This can sometimes mean they’re in a loop.
• Thread pool usage: are the thread pools over-utilized or under-utilized?
• Threads throwing exceptions.
• Several threads with identical stack traces. This can indicate that the program is looping and creating more and more unnecessary threads.
• Is the Finalizer thread moving? This processes a queue of objects waiting to be garbage collected, but containing a finalize() method that needs to be executed first. If this thread is slow-moving, it can result in memory errors.

Free Java Thread Dump Analyzers

There are several free tools available. JVM tools such as VisualVM have basic thread analysis capabilities, and some IDEs have plug-ins that can do simple thread analysis.

More comprehensive free tools include fastThread (free version),  Jstackreview and IBM Thread Monitor and Dump Analyzer (TDMA). IBM TDMA works best with IBM dump formats, but does perform limited analysis on thread dumps taken from other JVMs such as HotSpot.

Let’s look at a comparison chart of features offered by these free tools.

Paid Tools That Include Thread Dump Analysis

What’s available if we’re prepared to pay for a thread dump analyzer? The most popular tools in this category are the subscription version of fastThread, as well as the profiling tools JProfiler and YourKit. Both JProfiler and YourKit are multi-purpose rather than dedicated thread dump analyzers, but they do offer a good range of features for working with thread dumps. fastThread is a dedicated thread dump analyzer, and is also part of the yCrash suite, which offers a full range of troubleshooting and performance analysis tools.

The fastThread subscription version removes the limits on the number and size of dumps that can be submitted, adds an on-premise version for greater security, and adds more capability to the REST APIs available. It also includes complex machine-learning algorithms that perform deep analysis and provide recommendations.

Let’s compare the features offered by these three tools.

Conclusion

When an application isn’t running as it should, examining thread dumps provides a wealth of useful information  to isolate the problem. There are several excellent Java thread dump analyzer tools available, both free and paid. The most feature-rich of those we’ve looked at is fastThread, which is available as a free tool. A more comprehensive version is also available as subscription software.