There are 8 flavors of java.lang.OutOfMemoryError. In these 8 flavors
java.lang.OutOfMemoryError: unable to create new native thread
is one of the commonly occurring flavor. This type of ‘OutOfMemoryError’ is generated when an application isn’t able to create new threads. This error can surface because of following two reasons:
- There is no room in the memory to accommodate new threads.
- The number of threads exceeds the Operating System limit.
There are 6 potential solutions to address this
java.lang.OutOfMemoryError: unable to create new native thread
error. Depending on what event is triggering this error, either one or a combination of the below mentioned solutions can be applied to resolve the problem.
1. Fix Thread Creation Rate
When you see “java.lang.OutOfMemoryError: unable to create new native thread”, you should diagnose whether the application has started to create more threads. You can use online thread dump analyzer tool such as http://fastthread.io/ (which I would highly recommend), to see how many threads are created? What is the stack trace of those excessively created threads? Who is creating them? Once you know to these questions, it’s easy to solution them. Check out the ‘real world example’ section of this article, which walks through a ‘java.lang.OutOfMemoryError: unable to create new native thread’ problem experienced a major B2B travel application and how http://fastthread.io/ tool was used to diagnose the problem.
2. Increase the Thread Limits Set at Operating System
The Operating System has limits for the number of threads that can be created. The limit can be found by issuing
command. In certain servers, I have seen this value set to a low value such as 1024. It means totally only 1024 threads can be created in this machine. So if your application is creating more than 1024 threads, it’s going to run into “java.lang.OutOfMemoryError: unable to create new native thread”. In such circumstances increase this limit.
3. Allocate More Memory to the Machine
If you don’t see a high number of threads created and “ulimit –u” value is well ahead then it’s indicative that your application has grown organically and needed more memory to create threads. In such circumstance, allocate more memory to the machine. It should solve the problem.
4. Reduce Heap Space
One very important point that even seasoned engineers forget is: threads are not created within the JVM heap. They are created outside the JVM heap. So if there is less room left in the RAM, after the JVM heap allocation, application will run into “java.lang.OutOfMemoryError: unable to create new native thread”.
So let’s consider this example:
|Overall RAM size||6 GB|
|Heap size (i.e. –Xms and –Xmx)||5 GB|
|Perm Gen size (i.e. -XX:MaxPermSize and -XX:MaxPermSize)||512 MB|
As per this configuration 5.5 GB (i.e. 5 GB heap + 512 MB Perm Gen) is used by the JVM Heap and it leave only 0.5GB (i.e. 6 GB – 5.5GB) space. Note in this 0.5 GB space – kernel processes, other user processes and threads has to run. It may not be sufficient, and most likely the application will start to experience “java.lang.OutOfMemoryError: unable to create new native thread”.
To mitigate this problem, you can consider reducing the Heap Size from 5GB to 4GB (if your application can accommodate it without running into other memory bottlenecks).
5. Reduce Number of Processes
This solution is quite similar to ‘Reduce Heap Space’. Let’s looks into this scenario, where you are running multiple processes on a server which is constrained by memory. Say:
|Overall RAM size||32 GB|
|Number of Java Processes in the server||5|
|Heap size of each Java process||6 GB|
It means in total all of the java processes heap is occupying 30 GB (i.e. 5 processes X 6 GB) of memory. It leaves only 2 GB for kernel processes, other user processes and threads to run. It may not be sufficient, and most likely the application will start to experience “java.lang.OutOfMemoryError: unable to create new native thread”.
In this circumstance, it’s better to run only 4 java processes on one server. So that only 24 GB is occupied (4 processes X 6GB), and it leaves 8 GB (i.e. 32 GB – 24 GB) of memory. It might leave enough room for threads and run other processes to run.
6. Reduce Thread Stack Size (-Xss)
A thread occupies memory in RAM. So if each thread has high memory allocation, then overall memory consumption will also go higher. The default value of a thread’s memory size depends on the JVM provider. In some cases it’s 1mb. So if your application has 500 threads, then threads alone is going to occupy 500mb of space.
However, you can use the java system property –Xss to set the thread’s memory size. Using this property, you can throttle down the memory size. Example if you configure -Xss256k, your threads will only consume 125mb of space (i.e. 500 threads X 256k). So by lowering –Xss size also, you might be able to eliminate “java.lang.OutOfMemoryError: unable to create new native thread”
CAUTION: However if you configure –Xss to a very low value, you will start to experience java.lang.StackOverflowError. If you configure to even lower value, JVM will not even start.
Real World Example
Now let me walk through a real world example of “java.lang.OutOfMemoryError: unable to create new native thread”, which I diagnosed recently. This error was experienced by a major B2B travel application in North America. No recent production deployments were made to this application, but all of a sudden it started to throw “java.lang.OutOfMemoryError: unable to create new native thread”
Step 1: As an initial step, we captured the thread dump from the application when it was experiencing this “java.lang.OutOfMemoryError: unable to create new native thread”. Then we uploaded the thread dump into online thread dump analyzer http://fastthread.io/.
Step 2: http://fastthread.io/ tool reported that application had 3216 threads alive and rightly pointed that it can result in “java.lang.OutOfMemoryError: unable to create new native thread”. 3000+ threads was a very high thread count for this application, which is at least 6 times more than the regular period.
Fig 1: http://fastthread.io/ tool showing thread count summary
Step 3: Since now it’s confirmed that excessive threads are causing “java.lang.OutOfMemoryError: unable to create new native thread”, the next step is to identify what those excessively created threads are? And who is creating them? http://fastthread.io/ tool has a section “Repeating Stack traces”, in which threads with same stack traces are grouped together. In that section, we noticed that 2319 threads (i.e. 72%) are exhibiting same stack trace as shown in Fig 2.
Fig 2: http://fastthread.io/ tool showing group of threads which has same stack trace
Fig 3: Individual Thread’s stack trace as reported by the http://fastthread.io/ tool
From the stack trace, we inferred that these threads created are by the Datastax driver. This application uses DataStax driver for connecting with Apache Cassandra NoSQL Database. So now the question becomes all of sudden why Datastax driver started to create so many threads? No upgrades were made to this driver. No recent deployments were made to the application. Why all of a sudden this problem started?
Root cause: Apparently the problem turned out that Apache Cassandra NoSQL DB was running into disk space issue on one of its nodes. This issue caused the Datastax driver to spawn thousands of threads. Thus it cascaded as ‘java.lang.OutOfMemoryError: unable to create new native thread’ error on the JVM side. When more space was allocated to Apache Cassandra NoSQL DB nodes, the problem got resolved.