Category Archives: Articles in English

Cool, Tomcat is able to handle more than 13,000 concurrent connections.

Last time I have promised you to take a look at more real life scenario regarding threads. In the last blog entry I have shown that on modern operating system and JVM it’s not a problem to create 32,000 threads. Now I want to test how many threads can be handled by a Tomcat instance.

I just want to remind you the motivation. Some people believe that threads are expensive, that we should not create lot of them. They believe that it’s better to use different mechanisms like asynchronous servlets, specialized libraries etc. I just want to find out if we really need such measures or if good old threads are good enough.

If you read articles about asynchronous servlets, you find out that the main motivation is AJAX. Mainly the scenario, when a HTTP connection is open for a long time and the data are sent when an event occurs.

OK, let’s simulate it. We need to simulate lot of open HTTP connections waiting for an event. The easiest way to achieve it is my precious suicidal servlet.

public class ThreadsServlet extends HttpServlet {
	private static final long serialVersionUID = 7770323867448369047L;
	
	@Override
	protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
		int number = Integer.valueOf(req.getParameter("number"));
		try {
			System.out.println("Servlet no. "+number+" called.");
			URL url = new URL(req.getScheme()+"://"+req.getServerName()+":"+req.getServerPort()+req.getRequestURI()+"?number="+(number+1));
			Object content = url.getContent();
			resp.setContentType("plain/text");
			resp.getWriter().write("OK: "+content);
		} catch (Throwable e) {
			String message = "Reached "+number+" of connections";
			System.out.println(message);
			System.out.println(e);
			resp.getWriter().write(message);
		}
	}
}

The servlet is quite simple, it just opens HTTP connection to itself. So it basically tries to create infinite number of connections. Top keep track of the progress, there is a request parameter “number” that is incremented with each call. We can thus observe how many active connections we have.

Default configuration

Let’s run it. Just open “http://localhost:8080/threads/something?number=1” in your browser and see what happens.

Not much, in console (or logs/catalina.out) you can see

...
Servlet no. 37 called. 
Servlet no. 38 called. 
Servlet no. 39 called. 
Servlet no. 40 called.

What? Only 40 concurrent threads served? That’s not much. Let’s try better.

Connector configuration

We can reconfigure Tomcat connector to be able to serve more connections (server.xml)

    <Connector port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" 
               maxThreads="32000"/>

As we know from the last time, 32K is the OS limit, we can’t go over that. If we execute the test, the results are slightly better:

Servlet no. 485 called. 
Servlet no. 486 called. 
Servlet no. 487 called. 
Servlet no. 488 called. 
Servlet no. 489 called. 
Servlet no. 490 called. 
May 1, 2010 5:55:32 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run 
SEVERE: Socket accept failed 
java.net.SocketException: Too many open files 
        at java.net.PlainSocketImpl.socketAccept(Native Method) 
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:358) 
        at java.net.ServerSocket.implAccept(ServerSocket.java:470) 
        at java.net.ServerSocket.accept(ServerSocket.java:438) 
        at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61) 
        at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310) 
        at java.lang.Thread.run(Thread.java:636)

Wow, it looks like, that there is some limit on open files. Since I am not Linux guru, the first thing I have tried was to change Tomcat connector to nonblocking.

Nonblocking Connector

To use nonblocking connector, you have to set the protocol in server.xml

  <Connector port="8080" 
               connectionTimeout="20000" 
               redirectPort="8443" 
		protocol="org.apache.coyote.http11.Http11NioProtocol" 
		maxThreads="32000"/>

Unfortunately the result is almost the same:

Servlet no. 483 called. 
Servlet no. 484 called. 
Servlet no. 485 called. 
Servlet no. 486 called. 
May 1, 2010 5:59:24 PM org.apache.tomcat.util.net.NioEndpoint$Acceptor run 
SEVERE: Socket accept failed 
java.io.IOException: Too many open files 
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) 
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163) 
        at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198) 
        at java.lang.Thread.run(Thread.java:636)

Increase Open File Limit

Apparently, the blocking connector was not the issue. After some time spent with Google I have found the answer. Linux is limiting number of open connections. You can execute “ulimit -n” to see what’s your limit. Luckily, it’s possible to change the limit. You can either set it by “ulimit -n 32768” if you have permissions or by adding following lines to /etc/security/limits.conf (lukas is my username)

lukas            hard    nofile          32768 
lukas            soft    nofile          32768

To apply this change you have to logout and login. After that, you will see this:

Servlet no. 5856 called. 
Servlet no. 5857 called. 
Servlet no. 5858 called. 
Servlet no. 5859 called. 
May 1, 2010 6:07:58 PM org.apache.tomcat.util.net.NioEndpoint$SocketProcessor run 
SEVERE: 
java.lang.OutOfMemoryError: GC overhead limit exceeded 
	at java.util.Arrays.copyOf(Arrays.java:2894) 
	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:117) 
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:407) 
	at java.lang.StringBuilder.append(StringBuilder.java:136) 
	at java.lang.StringBuilder.append(StringBuilder.java:132) 
	at java.lang.Throwable.printStackTrace(Throwable.java:529) 
	at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:94) 
	at java.util.logging.StreamHandler.publish(StreamHandler.java:196) 
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105) 
	at java.util.logging.Logger.log(Logger.java:476) 
	at java.util.logging.Logger.doLog(Logger.java:498) 
	at java.util.logging.Logger.logp(Logger.java:698) 
	at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:167) 
	at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:135) 
	at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:755) 
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2080) 
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) 
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) 
	at java.lang.Thread.run(Thread.java:636)

We have reached 5856 threads before we run out of memory. Cool.

Increase Heap Size

Let’s try to increase the heap size. Just add “-Xmx2048m” to JAVA_OPTS. Before I have started running out of heap, I got to cca 11000 threads! Is it enough? I do not know, but I think it’s pretty good.

Moreover, if you do a heap dump, you will see, that most of the memory is consumed by char and byte arrays. (This heap dump has been taken with approximately 5000 connections, screenshot from VisualVM)

Heap Dump

It’s understandable, we have lot of open buffers on both sides. I assume that we have at least four buffers per servlet. One for sevlet request, one for servlet response, one for URL request and one for URL response. But maybe there will be other buffers as well. To be honest, I have to admit that memory consumed by stacks would not appear here, it’s probably handled by OS. But we have run out of the heap, so that’s why I am talking about it.

Smaller buffers

We can try to make some of the buffers smaller. I was able to find only one setting that had some effect. Again it is connector setting in server.xml config file.

  <Connector port="8080" 
               connectionTimeout="200000" 
               redirectPort="8443" 
		protocol="org.apache.coyote.http11.Http11NioProtocol" 
		maxThreads="32000" 
		socket.appReadBufSize="1024" 
		socket.appWriteBufSize="1024" 
		bufferSize="1024"/>

With this setting, I was able to get near to 13000 open connections.

...
Servlet no. 13327 called.
Servlet no. 13328 called.
Servlet no. 13329 called.
Servlet no. 13330 called.
Servlet no. 13331 called.
Servlet no. 13332 called.

After that the machine started to run out of physical memory, GC took ages so I had to stop the server. (Just to remind you, my test machine is two year old laptop with Intel Core 2 Duo T8100 2.1GHz with 4GB of RAM. There is 64bit Linux 2.6.32 and OpenJDK (IcedTea6 1.8) running on top of it.)

As we have seen, threads are not the major issue on modern machines. There is probably significant amout of memory consumed by the stack traces too, but I think the biggest problem are the buffers. And the important point is, that we would need the buffers even if we used asynchronous servlets! Of course, there is still some overhead connected with threads, so asynchronous libraries have their place. In fact, it would be nice to try similar experiment with asynchronous servlets. I am afraid, that I will not be able to do it, but I will be glad to help if there is some volunteer.

Please also note that your numbers may vary. After all this has been quite artificial test. I think it’s simulates lot of real-life use cases, but you know, the reality is always different.

On the other hand, with more physical memory and better Tomcat configuration, we might got to higher numbers. I have heard legends about 16K threads.

I think that I will finish with my favorite message. Do not use complicated constructs unless you are sure you need them. Please remember golden rules of optimization:

The First Rule of Program Optimization: Don’t do it.
The Second Rule of Program Optimization (for experts only!): Don’t do it yet.

If you want to verify my results, the source code is here. If you have some comments, different results or advices, do not hesitate to add a comment.

Resources:
Tomcat connector config

Why we need asynchronous servlets

Note: If you wonder why the hell I have started to write in something that looks almost like English when apparently I do even more mistakes than in Czech, the answer is simple. I just need to practice my English (apart from that I want to be world famous, not just known in Czech Republic)

How many threads a JVM can handle?

Threads are expensive. They need lot of system resources, it’s time-consuming to create them and JVM is able to manage only few of them. Therefore we need asynchronous processing, new libraries, new languages etc. At least, that’s what I am hearing from everywhere. But is it really true? Are threads really so expensive? There is only one way how to find out. Just ask the machine.

The only thing we need to do is write a simple test. We just have to create as many sleeping or waiting threads as possible and we will see when it breaks. The reason why we are interested only in waiting threads is obvious. It has no sense to create thousands of active threads if we have only few CPUs. We want to simulate a situation, when threads are waiting for database data, incoming message or other CPU unrelated tasks.

public class CreateThreads {
	/**
	 * Thread class
	 * @author Lukas Krecan
	 *
	 */
	private static final class MyThread extends Thread
	{
		private final int number;
	
		public MyThread(int number) {
			this.number = number;
		}
		@Override
		public void run() {
			if (shouldPrintMessage(number))
			{
				System.out.println("Thread no. "+number+" started.");
			}
			try {
				//sleep forever
				Thread.sleep(Long.MAX_VALUE);
			} catch (InterruptedException e) {
				e.printStackTrace();
			}
		}
	}
	/**
	 * @param args
	 */
	public static void main(String[] args) {
		long startTime = System.currentTimeMillis(); 
		final int noOfThreads = 1000000;
		int i=0;
		try {
			for (i=0; i < noOfThreads; i++)
			{
				if (shouldPrintMessage(i))
				{
					System.out.println("Creating thread "+i+" ("+(System.currentTimeMillis()-startTime)+"ms)");
				}
				new MyThread(i).start();
			}
		} catch (Throwable e) {
			System.out.println("Error thrown when creating thread "+i);
			e.printStackTrace();
		}
	}
	
	private static boolean shouldPrintMessage(int i)
	{
		return i % 100 == 0;
	}
}

As you can see, there is nothing special in the test. Every hundredth threads prints a message to console and all of them sleep forever. In the main method we are creating threads and when an exception is thrown, number of created threads is printed out.

Even though there is nothing special on the test, the results are really interesting and for me quite surprising. I will let you guess the results. The test is executed on my two years old laptop with Intel Core 2 Duo T8100 2.1GHz. There is 64bit Linux 2.6.31 and OpenJDK (IcedTea6 1.6.1) running on top of it.

Try to guess how many threads are created if I do not tinker with JVM arguments at all. Is it 1K? Or 10K? Or even more? Well, it's slightly more than 32K. Usually you see result like this

Creating thread 31900 (37656ms) 
Thread no. 31900 started. 
Creating thread 32000 (37736ms) 
Thread no. 32000 started. 
Creating thread 32100 (37818ms) 
Thread no. 32100 started. 
Error thrown when creating thread 32172 
java.lang.OutOfMemoryError: unable to create new native thread 
	at java.lang.Thread.start0(Native Method) 
	at java.lang.Thread.start(Thread.java:614) 
	at CreateThreads.main(CreateThreads.java:43)

The question is, why we can not create more threads. Are we constrained by the heap size? Or maybe stack size? No in fact, 32K is apparently Linux kernel limit. Therefore there is nothing we can do in Java, Linux is just not able to handle more threads. But I think that 32K threads is not bad at all. Please keep in mind, that you will be able to reproduce the results only on modern operating systems and JVMs. I am afraid that Windows XP results will be much worse.

Please also note, that once the threads are created and all of them sleep, the application does not consume any CPU and it consumes around 800MB of memory. I am not sure why so much memory is used, it has to be investigated.

As we have seen, in this artificial scenario threads are relatively cheap. I do not want to say, that we should not use new libraries or tools. Most of them are quite useful. The only thing I want to say is that we should think twice before prematurely optimizing our applications. Sometimes the most straightforward and easiest approach is the best.

Next time, I will try to do some more real life example with servlet container.

Test coverage – interesting only when low

What a nice and controversial title. But I really mean it. Level of test coverage is really interesting only when it’s low. Of course, we should aim for highest coverage possible, but the higher the coverage is, the less useful information it brings. Let take the following example and let’s assume there are no tests for this method.

	/**
	 Adds positive numbers. If one of the parameters is negative,
	 IllegalArgumentException is thrown.
	 * @param a
	 * @param b
	 * @return
	 */
	public int addPositive(int a, int b)
	{
		if ((a<0) || (b<0))
		{
			throw new IllegalArgumentException("Argument is negative");
		}
		return a+a;
	}

No coverage - wow, that's interesting

The fact that some piece of code is not covered at all is really interesting. It either means, that we have to start writing tests at once or that the piece of code has to be deleted. Either way we know that we should do something. Moreover, we know exactly what to do. For example, we can start with the following test.

	@Test
	public void testAddition()
	{
		assertEquals(2, math.addPositive(1, 1));
	}

It will get us partial coverage like this

Partial coverage - cool, I know what to test next

If I have partial coverage, I can see what other tests I am supposed to write. In our example I see that I have to test what happens if I pass in a negative numbers.

	@Test(expected=IllegalArgumentException.class)
	public void testNegative1()
	{
		math.addPositive(-1, 1);
	}
	@Test(expected=IllegalArgumentException.class)
	public void testNegative2()
	{
		math.addPositive(1, -1);
	}

Total coverage - I have no clue

Now we have 100% coverage. Well done! Unfortunately, I have no clue what to do next. I have no idea if I need to write more tests, I do not know if I have complete functional coverage. I have to stop looking at the coverage and I have to start thinking. You see, when the coverage was low it gave me lot of useful information, when it's high I am left on my own. For example I can think hard and come up with following complex test scenario.

	@Test
	public void testComplicatedAddition()
	{
		assertEquals(3, math.addPositive(1, 2));
	}

Please note, that the coverage was already 100% before I have written the test. So one more test could not make it higher. Nevertheless the test actually detected a bug that was there from the beginning. It means that 100% code coverage is not enough, we have to aim higher. We should care about functional coverage and not about code coverage!

That's also one of many reasons why we should do test driven development. Or at least test first development. If we write the tests before the code, we are forced to think from the beginning. In TDD we do not care about the coverage. We just think about the functionality, write corresponding tests and then add the code. Total coverage is automatically achieved.

In the opposite situation, when we write the code first and then add the tests as an afterthought, we are not forced to think. We can just write some tests, achieve the target coverage and be happy. Test coverage can be really deceitful. It can give us false sense of security. So please remember, test coverage is a good servant but a bad master.