There are 8 flavors of java.lang.OutOfMemoryError. In these 8 flavors

java.lang.OutOfMemoryError: unable to create new native thread

is one of the commonly occurring flavor. This type of

OutOfMemoryError

is generated when an application isn’t able to create new threads. This error can surface because of following two reasons:

  1. There is no room in the memory to accommodate new threads.
  2. The number of threads exceeds the Operating System limit.

Solutions

There are 6 potential solutions to address this

java.lang.OutOfMemoryError: unable to create new native thread

error. Depending on what event is triggering this error, either one or a combination of the below mentioned solutions can be applied to resolve the problem.

1. Fix Thread Creation Rate

When you see

java.lang.OutOfMemoryError: unable to create new native thread,

you should diagnose whether the application has started to create more threads. You can use online thread dump analyzer tool such as http://fastthread.io/ (which I would highly recommend), to see how many threads are created? What is the stack trace of those excessively created threads? Who is creating them? Once you know to these questions, it’s easy to solution them. Check out the ‘real world example’ section of this article, which walks through a ‘

java.lang.OutOfMemoryError: unable to create new native thread’

problem experienced a major B2B travel application and how http://fastthread.io/ tool was used to diagnose the problem.

2. Increase the Thread Limits Set at Operating System

The Operating System has limits for the number of threads that can be created. The limit can be found by issuing “

ulimit –u

” command. In certain servers, I have seen this value set to a low value such as 1024. It means totally only 1024 threads can be created in this machine. So if your application is creating more than 1024 threads, it’s going to run into

java.lang.OutOfMemoryError: unable to create new native thread.

In such circumstances increase this limit.

3. Allocate More Memory to the Machine

If you don’t see a high number of threads created and

ulimit –u

value is well ahead then it’s indicative that your application has grown organically and needed more memory to create threads. In such circumstance, allocate more memory to the machine. It should solve the problem.

4. Reduce Heap Space

One very important point that even seasoned engineers forget is: threads are not created within the JVM heap. They are created outside the JVM heap. So if there is less room left in the RAM, after the JVM heap allocation, application will run into

java.lang.OutOfMemoryError: unable to create new native thread.

So let’s consider this example:

Overall RAM size 6 GB
Heap size (i.e. –Xms and –Xmx) 5 GB
Perm Gen size (i.e. -XX:MaxPermSize and -XX:MaxPermSize) 512 MB

As per this configuration 5.5 GB (i.e. 5 GB heap + 512 MB Perm Gen) is used by the JVM Heap and it leave only 0.5GB (i.e. 6 GB – 5.5GB) space. Note in this 0.5 GB space – kernel processes, other user processes and threads has to run. It may not be sufficient, and most likely the application will start to experience

java.lang.OutOfMemoryError: unable to create new native thread

To mitigate this problem, you can consider reducing the Heap Size from 5GB to 4GB (if your application can accommodate it without running into other memory bottlenecks).

5. Reduce Number of Processes

This solution is quite similar to ‘Reduce Heap Space’. Let’s looks into this scenario, where you are running multiple processes on a server which is constrained by memory. Say:

Overall RAM size 32 GB
Number of Java Processes in the server 5
Heap size of each Java process 6 GB

It means in total all of the java processes heap is occupying 30 GB (i.e. 5 processes X 6 GB) of memory. It leaves only 2 GB for kernel processes, other user processes and threads to run. It may not be sufficient, and most likely the application will start to experience

java.lang.OutOfMemoryError: unable to create new native thread

In this circumstance, it’s better to run only 4 java processes on one server. So that only 24 GB is occupied (4 processes X 6GB), and it leaves 8 GB (i.e. 32 GB – 24 GB) of memory. It might leave enough room for threads and run other processes to run.

6. Reduce Thread Stack Size (-Xss)

A thread occupies memory in RAM. So if each thread has high memory allocation, then overall memory consumption will also go higher. The default value of a thread’s memory size depends on the JVM provider. In some cases it’s 1mb. So if your application has 500 threads, then threads alone is going to occupy 500mb of space.

However, you can use the java system property –Xss to set the thread’s memory size. Using this property, you can throttle down the memory size. Example if you configure -Xss256k, your threads will only consume 125mb of space (i.e. 500 threads X 256k). So by lowering –Xss size also, you might be able to eliminate

java.lang.OutOfMemoryError: unable to create new native thread

CAUTION: However if you configure –Xss to a very low value, you will start to experience java.lang.StackOverflowError. If you configure to even lower value, JVM will not even start.

Real World Example

Now let me walk through a real world example of

java.lang.OutOfMemoryError: unable to create new native thread

which I diagnosed recently. This error was experienced by a major B2B travel application in North America. No recent production deployments were made to this application, but all of a sudden it started to throw

java.lang.OutOfMemoryError: unable to create new native thread

Step 1: As an initial step, we captured the thread dump from the application when it was experiencing this

java.lang.OutOfMemoryError: unable to create new native thread

Then we uploaded the thread dump into online thread dump analyzer http://fastthread.io/.

Step 2: http://fastthread.io/ tool reported that application had 3216 threads alive and rightly pointed that it can result in

java.lang.OutOfMemoryError: unable to create new native thread

3000+ threads was a very high thread count for this application, which is at least 6 times more than the regular period.

Image title
Fig 1: http://fastthread.io/ tool showing thread count summary

Step 3: Since now it’s confirmed that excessive threads are causing

java.lang.OutOfMemoryError: unable to create new native thread

the next step is to identify what those excessively created threads are? And who is creating them? http://fastthread.io/ tool has a section “Repeating Stack traces”, in which threads with same stack traces are grouped together. In that section, we noticed that 2319 threads (i.e. 72%) are exhibiting same stack trace as shown in Fig 2.

Image title
Fig 2: http://fastthread.io/ tool showing group of threads which has same stack trace
Image title
Fig 3: Individual Thread’s stack trace as reported by the http://fastthread.io/ tool

From the stack trace, we inferred that these threads created are by the Datastax driver. This application uses DataStax driver for connecting with Apache Cassandra NoSQL Database. So now the question becomes all of sudden why Datastax driver started to create so many threads? No upgrades were made to this driver. No recent deployments were made to the application. Why all of a sudden this problem started?

Root cause: Apparently the problem turned out that Apache Cassandra NoSQL DB was running into disk space issue on one of its nodes. This issue caused the Datastax driver to spawn thousands of threads. Thus it cascaded as ‘java.lang.OutOfMemoryError: unable to create new native thread’ error on the JVM side. When more space was allocated to Apache Cassandra NoSQL DB nodes, the problem got resolved.

Advertisements