Description

If threads are blocking momentarily, then it’s not a problem. However, if they are blocking for a prolonged period, then it’s of concern. It’s indicative of some problem in the application. Typically blocked threads would make application unresponsive.

Threads that remain in the same method and in ‘BLOCKED’ state between 3 threads dump which are captured in an interval of 10 seconds can turn out to be problematic ones. Studying the stack traces of such blocked threads would indicate the reasons why they are blocked. Reasons may include: deadlocks, circular deadlocks, another thread, would have acquired the locked and never released it, external SORs could have become unresponsive …

Example

Following is the excerpt of a thread dump that was captured from a major SOA application, which became unresponsive. The thread ajp-bio-192.168.100.128-9022-exec-173 remained in BLOCKED state for 3 consecutive thread dumps which were captured in an interval of 10 seconds. Here goes the important parts of Stack  trace of this thread:

ajp-bio-192.168.100.128-9022-exec-173 
Stack Trace is: 
java.lang.Thread.State: BLOCKED (on object monitor)
at **.***.sp.dao.impl.ReferenceNumberGeneratorDaoImpl.getNextItineryReferenceNumber(ReferenceNumberGeneratorDaoImpl.java:55)
- waiting to lock 0x00000006afaa5a60 (a **.***.sp.dao.impl.ReferenceNumberGeneratorDaoImpl)
at sun.reflect.GeneratedMethodAccessor3112.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
:
:
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

Here you can notice that ajp-bio-192.168.100.128-9022-exec-173 stuck in the method **.***.sp.dao.impl.ReferenceNumberGeneratorDaoImpl.getNextItineryReferenceNumber(ReferenceNumberGeneratorDaoImpl.java:55). This thread got stuck in this method, because another thread ajp-bio-192.168.100.128-9022-exec-84 after obtaining the lock 0x00000006afaa5a60, it never returned back. Below goes the stack trace of ajp-bio-192.168.100.128-9022-exec-84 thread

ajp-bio-192.168.100.128-9022-exec-84 
Stack Trace is: 
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at net.sourceforge.jtds.jdbc.SharedSocket.readPacket(SharedSocket.java:850)
at net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket(SharedSocket.java:731)
- locked 0x00000006afa88a68 (a java.util.concurrent.ConcurrentHashMap)
at net.sourceforge.jtds.jdbc.ResponseStream.getPacket(ResponseStream.java:477)
at net.sourceforge.jtds.jdbc.ResponseStream.read(ResponseStream.java:114)
at net.sourceforge.jtds.jdbc.ResponseStream.peek(ResponseStream.java:99)
at net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:4127)
at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1086)
- locked 0x00000006bca709f8 (a net.sourceforge.jtds.jdbc.TdsCore)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:493)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:1032)
at com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:174)
at **.***.sp.dao.impl.ReferenceNumberGeneratorDaoImpl.getNextItineryReferenceNumber(ReferenceNumberGeneratorDaoImpl.java:65)
- locked 0x00000006afaa5a60 (a **.***.sp.dao.impl.ReferenceNumberGeneratorDaoImpl)
at sun.reflect.GeneratedMethodAccessor3112.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
:
:
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

Apparently, it turned out that JDBC read timeout was never set, due to that ajp-bio-192.168.100.128-9022-exec-84 never returned back from it’s JDBC call. Thus it blocked all other threads which were invoking **.***.sp.dao.impl.ReferenceNumberGeneratorDaoImpl.getNextItineryReferenceNumber() method.

Why named as Atherosclerosis?

Atherosclerosis is a heart disease. Medically it’s defined as the following: the inside walls of human arteries are normally smooth and flexible, allowing blood to flow through them easily. Fatty deposits, or plaques, may build up inside the arterial wall. These plaques narrow the artery and can reduce or even completely stop the flow of blood, leading to death.

Similarly if blocking of a thread prolongs and happens across multiple threads, then it would make the application unresponsive, eventually, it has to be killed.

Advertisements