A travel organization in North America encountered unresponsive microservices because of too many threads. The Site Reliability Engineering (SRE) team analyzed thread dumps and found that 2,319 threads were stuck waiting for network responses due to an issue with a Cassandra database. Fixing a disk space shortage restored normal performance and helped prevent future problems. This approach was essential for quick and effective problem-solving.
A major financial institution in North America experienced serious outages in its middleware application, which is essential for banking services. The Site Reliability Engineering (SRE) team analyzed thread dumps and found that a bug in the Oracle JDBC driver was causing too many active threads. Updating to a patched driver fixed the issue, stabilized operations, and helped prevent future outages.
Laurentio Spilca, a Java and Spring expert, authored "Troubleshooting Java" to teach efficient code investigation, optimization, and bug fixing. The book includes techniques from basic debugging to advanced microservices methods. Notably, it demonstrates using the yCrash-fastThread tool for analyzing thread dumps, effectively diagnosing deadlock issues in Java applications.
The application connecting to the Oracle RAC cluster experienced performance issues due to resource constraints, causing unresponsiveness. Troubleshooting artifacts were collected and analyzed using yCrash, revealing that 46% of threads were blocked waiting for database responses. After escalating to the DBA, resource adjustments resolved the slowdown, restoring normal application function.
Apache PDFBox is an open-source library for handling PDF documents in Java. A recent deadlock issue was identified in the library, which occurs when two threads are blocked, waiting for resources held by each other. The problem was diagnosed using a thread dump, and a ticket was filed, resulting in a prompt fix by the PDFBox team.
The post discusses troubleshooting intermittent HTTP 502 Bad Gateway errors in an application running on AWS Elastic Beanstalk. The errors were traced to the Tomcat server, which was terminated due to insufficient RAM (1GB). Upgrading to a 2GB RAM EC2 instance resolved the issue, eliminating the errors.
Java developers often use 'java.util.UUID#randomUUID()' to generate UUIDs, but it can hinder application availability due to potential 'entropy' shortages in the operating system. This can lead to multiple threads entering a BLOCKED state, causing application unresponsiveness. Solutions include upgrading RHEL, installing 'haveged,' or using '/dev/urandom' for randomness.
On October 11, 2021, GCeasy experienced an outage due to HTTP 504 errors caused by inefficiencies in a new SQL query from a recent deployment. AWS CloudWatch monitored CPU and database connection spikes, while yCrash identified the problematic code. Commenting out the SQL query restored application performance.
we troubleshooted a CPU spike problem that surfaced in a major trading application in North America. All of a sudden, this application’s CPU started to spike up to 100%