Java crumbs » Articles in English

Writing fluent API

Lukáš Křečan — Sat, 10 Jul 2010 13:12:19 +0000

In recent weeks I have been working with Arjen Poutsma on a brand new testing support for Spring Web Services. At least Arjen has been working, I have been just generating crazy ideas and he kept explaining me why we can't use them.

But it was quite interesting experience and I have learned a lot about fluent API design. Since it was my first fluent API and I am not an API expert, please take everything I mention here with grain of salt.

The task is simple. We need a way how to configure a mock. This mock has to validate that requests generated by our application are correct and after that the mock has to return some response. We want to write something like this:

Computer, when someone is trying to send a request, verify that the request contains value “a”, has header “header” and then respond with value “b”.

The requirements for the API are following:

It has to be simple for users. Mainly it has to be friendly to GUI addicts like me, that rely on code-completion all the time.
The library itself has to be simple
The API should be as fluent as possible
It has to be extensible

In the next part, I will try to describe several variants of the solution, each of them fulfilling the requirements differently. Please keep in mind that the code is simplified, it focuses solely on the API, not on the implementation. Moreover, the classes have been renamed to show what their main purpose is.

All characters appearing in this work are fictitious. Any resemblance to real persons, living or dead, is purely coincidental.

Interface based version

code
The first variant is based mainly on interfaces. The test itself looks like this

	@Test
	public void testMock()
	{
		Mock mock = new Mock();
		mock.whenConnecting().expectValue("a").and().expectHeader("header").andRespondWithValue("b");

		//test code

		mock.verify();
	}

There is one entry point, class Mock

public class Mock {

	public RequestExpectations whenConnecting()
	{
		return new MockInternal();
	}

	public void verify()
	{
		//do something
	}
}

It just returns an internal class that implements RequestExpectations interface.

public interface RequestExpectations {

    ResponseActions expectValue(String value);

    ResponseActions expectHeader(String name);
}

All methods of this interface return ResponseActions interface

public interface ResponseActions {

	RequestExpectations and();

	void andRespondWithValue(String value);

	void andRespondWithError();

}

Please note that and() method allows chaining of request expectation.

This API is really IDE friendly. Code completion works like a charm. It's quite simple both from outside and inside. But this API has one big flaw. It's not extensible. Imagine that we would like to add an expectation that verifies a digital signature. We would have to add a method to the RequestExpectations interface. It is not possible neither for third parties nor for us, since this change would break backward compatibility.

Static factory

code
We can split the functionality and allow others to provide their own validators and response generators. We just have to change the interfaces

public interface RequestExpectations {

    ResponseActions expect(RequestMatcher matcher);
}

and

public interface ResponseActions {

	RequestExpectations and();

	void andRespond(ResponseCallback callback);
}

Now we have made the API extensible, but we have to provide an easy way how to create RequestMatchers and ResponseCallbacks. One way how to do it is a statically imported static factory.

public class StaticFactory {
	public static RequestMatcher value(String value)
	{
		return new RequestMatcher() {
			public void match(Object someParams) {
				//do something
			}
		};
	}
	public static ResponseCallback withValue(String value)
	{
		return new ResponseCallback() {
			public void doWithResponse(Object someParams) {
				//do something
			}
		};
	}
}

The test then can look like this.

import static net.javacrumbs.fluentapi2.StaticFactory.*;

public class MockTest {

	@Test
	public void testMock()
	{
		Mock mock = new Mock();
		mock.whenConnecting().expect(value("a")).and().expect(header("header")).andRespond(withValue("b"));

		//test code

		mock.verify();
	}
}

Such API is also fluent. It even resembles EasyMock API. The issue here is that code completion does not work as well as before. In Eclipse you have to type first letter of the method name before Ctrl+Space starts working. In Idea you have to press Ctrl+Shift+Space. But the advantage is that everyone can provide their own factory and extend the library seamlessly and consistently.

Super-static variant

code
We can go event further. We can get rid of RequestExpectations interface, remove Mock class and do statically most of the stuff. The test can look like this

import static net.javacrumbs.fluentapi3.StaticFactory.*;

public class MockTest {

	@Test
	public void testMock()
	{
		expect(value("a")).andExpect(header("header")).andRespond(withValue("b"));

		//test code

		verify();
	}
}

It's simple and elegant. We even got rid of the strange and() method. The downside is that we usually have to use some ThreadLocals internally. But it looks almost like EasyMock. Cool.

Inheritance

code
Some people (like me) do not like static methods. It does not feel right, it's not object oriented programming. Can we do something similar, but dynamic? Yes, we can.

We just have to use inheritance. Just delete static keywords from the methods in StaticFactory class. We will also rename the class itself because it's not static any more.

We get

public class MockTest extends MockFactory{

	@Test
	public void testMock()
	{
		expect(value("a")).andExpect(header("header")).andRespond(withValue("b"));

		//test code

		verify();
	}
}

The downside is, that we have to extend our class from MockFactory. Sometimes it's not possible. No problem, we can use a trick I have seen in Apache Camel DSL.

public class MockTest{
	@Test
	public void testMock()
	{
		MockFactory mockFactory = new MockFactory()
		{
			protected void configure() {
				expect(value("a")).andExpect(header("header")).andRespond(withValue("b"));
			};
		};

		//test code

		mockFactory.verify();
	}
}

It's not as elegant as the direct inheritance, but it works. If we are brave enough, we can even use a closure.

public class MockTest extends MockFactory{
	@Test
	public void testMockClosure()
	{
		MockFactory mockFactory = new MockFactory()
		{{
				expect(value("a")).andExpect(header("header")).andRespond(withValue("b"));
		}};

		//test code

		mockFactory.verify();
	}
}

(I bet you did not know that Java has closures already).

So I have shown you four variants, which one is the best? I do not know, each of them has it's pros and cons. I personally like the last one the most. But the super static variant is not bad neither. Or the static? Or some mix between them? Some other? You have to make your own opinion.

References:
http://martinfowler.com/bliki/FluentInterface.html
http://martinfowler.com/dslwip/InternalOverview.html
http://camel.apache.org/routes.html
http://easymock.org/

Spring WS Security on both client and server

Lukáš Křečan — Mon, 07 Jun 2010 11:17:13 +0000

Recently, I have been playing with Spring WS with WS-Security. I just want to write down how it works. Do not except anything special, just simple example of basic security operations.

The example

We want to implement both client and server side. The client will sign the message, encrypt some part of it and add a timestamp. To make it more complex and real-life like we will sign the message using private key with alias “client” and encrypt the message using public key called “server”. Server will validate that the request is valid and will just sign the response using his key called “server”. Please note that I have picked Wss4j implementation because the configuration seemed to be easier than Xws.

Client

It's easy to do configure client interceptor like this (full configuration).

...

As you can see, there is nothing special. We just define which actions to take and properties. The only confusing part is, that key alias is defined as “securementUsername”.

Whit this configuration we will get following SOAP message.


	
		
			
				
				
					
						
							
								CN=Test Server,OU=Test
								1275904530
							
						
					
				
				
					fwFM7ShJ1xd7dTGrkh0410sTmW92OPB1q1fpzB21XFIe36siDDJWGgbw5B94yjmGK2YaPOWLb7cpVTYPzc9VUDs7Jc42CtrhT2H6eZ7CDiA60Ugz+qi2UyyfMDK6Vrdj9J68rij5P12AiBeTnd2wlhI29+71XbUpD5weHDHjMtQ=
					
				
				
					
				
			
			
				
					
					
					
						
							
						
						
						AU9utUgz5RylYCRDUAO0JWM48kM=
					
				
				
					NHjjgpb9/alUOq50CqPKLcdYrp7edYdKJDNvIhh+2OAhYdDvZmD1qGsVKd1H9oKPF17uaF2Sv3aY
					0le6BrvzVx3n2+nYYlHwAWlzBk7wsBt4vLll6q6juLCP+siupTIb1PeZDf3WrAbHUQh5oqjD6cZB
					Sc89pDspWRABQ8wPxYE=

				
					
						
							
								CN=Lukas Krecan,OU=Test
								1275900789
							
						
					
				
			
			
				2010-06-07T11:03:35.749Z
				2010-06-07T11:08:35.749Z
			
		
	
	
		
			
				
					
					
						
							
						
					
					
						81TEtUhHXo6iZeAmYrtYlm2ObAqOBpjfzf2VOVUg4Hs=
						
					
				
			
			2

Server config

To configure server, you have to define Spring WS server interceptor like this (full example).

No surprise here neither. The response will look like this.


	
		
			
				
					
					
					
						
							
						
						
						hEdDfxM6Nfs62Jxe8EOsELCDtUk=
					
					
						
							
						
						
						TTSRri5KJqXeMJfjzXyVmUewPxc=
					
				
				
					V5by3bOoGQNajfs7i9xQ+cbAqIkI0NS9N9FQlLb/dAuQfguE7jKRP9iypOeRLHCPr7g3BNg+NCrX
					6YcgDQ0TfXNhdL00AmoEfDmWSNvIVNE49kZEn3Ji/RW4VtdEiV79VD7Vuay0YAYGo9DSQvzq3FP6
					YEhfzfMqvfbWMdEKcO8=

				
					
						
							
								CN=Test Server,OU=Test
								1275904530
							
						
					
				
			
			
		
	
	
		
			3

As we have seen it's possible to configure WS-Security without much hassle. To learn more, visit the official Spring WS reference. You can download full example here.

More than 16,000 connections in Tomcat

Lukáš Křečan — Thu, 06 May 2010 12:19:27 +0000

This is just a short update. Last time, I have reached around 13,000 concurrent connections in Tomcat. Based on some comments (thanks IA), I have updated the test.

The most important change is that HttpClient is used instead of standard HTTP connection. Moreover the servlet address is hard-coded, so a new String is not created every time. I do not believe that this change has big impact but the code generates less garbage so GC has easier job.

With this small enhancements, I was able to get more than 16,000 connections running

...
Servlet no. 16354 called.
Servlet no. 16355 called.
Servlet no. 16356 called.
Servlet no. 16357 called.
May 6, 2010 1:21:54 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: I/O exception (java.net.SocketException) caught when processing request: Too many open files
May 6, 2010 1:21:54 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry

Again, we get "To many open files". The limit is 32768 and we have two open connections per thread (incoming/outgoing). Theoretically, we could get even higher, but the heap was still filling really fast, so GC was quite slow. And again, threads are not the main issue, the problem is still somewhere else.

The source code is available here.

Cool, Tomcat is able to handle more than 13,000 concurrent connections.

Lukáš Křečan — Sun, 02 May 2010 09:37:35 +0000

Last time I have promised you to take a look at more real life scenario regarding threads. In the last blog entry I have shown that on modern operating system and JVM it's not a problem to create 32,000 threads. Now I want to test how many threads can be handled by a Tomcat instance.

I just want to remind you the motivation. Some people believe that threads are expensive, that we should not create lot of them. They believe that it's better to use different mechanisms like asynchronous servlets, specialized libraries etc. I just want to find out if we really need such measures or if good old threads are good enough.

If you read articles about asynchronous servlets, you find out that the main motivation is AJAX. Mainly the scenario, when a HTTP connection is open for a long time and the data are sent when an event occurs.

OK, let's simulate it. We need to simulate lot of open HTTP connections waiting for an event. The easiest way to achieve it is my precious suicidal servlet.

public class ThreadsServlet extends HttpServlet {
	private static final long serialVersionUID = 7770323867448369047L;

	@Override
	protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
		int number = Integer.valueOf(req.getParameter("number"));
		try {
			System.out.println("Servlet no. "+number+" called.");
			URL url = new URL(req.getScheme()+"://"+req.getServerName()+":"+req.getServerPort()+req.getRequestURI()+"?number="+(number+1));
			Object content = url.getContent();
			resp.setContentType("plain/text");
			resp.getWriter().write("OK: "+content);
		} catch (Throwable e) {
			String message = "Reached "+number+" of connections";
			System.out.println(message);
			System.out.println(e);
			resp.getWriter().write(message);
		}
	}
}

The servlet is quite simple, it just opens HTTP connection to itself. So it basically tries to create infinite number of connections. Top keep track of the progress, there is a request parameter “number” that is incremented with each call. We can thus observe how many active connections we have.

Default configuration

Let's run it. Just open “http://localhost:8080/threads/something?number=1” in your browser and see what happens.

Not much, in console (or logs/catalina.out) you can see

...
Servlet no. 37 called.
Servlet no. 38 called.
Servlet no. 39 called.
Servlet no. 40 called.

What? Only 40 concurrent threads served? That's not much. Let's try better.

Connector configuration

We can reconfigure Tomcat connector to be able to serve more connections (server.xml)

As we know from the last time, 32K is the OS limit, we can't go over that. If we execute the test, the results are slightly better:

Servlet no. 485 called.
Servlet no. 486 called.
Servlet no. 487 called.
Servlet no. 488 called.
Servlet no. 489 called.
Servlet no. 490 called.
May 1, 2010 5:55:32 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:358)
        at java.net.ServerSocket.implAccept(ServerSocket.java:470)
        at java.net.ServerSocket.accept(ServerSocket.java:438)
        at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
        at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)
        at java.lang.Thread.run(Thread.java:636)

Wow, it looks like, that there is some limit on open files. Since I am not Linux guru, the first thing I have tried was to change Tomcat connector to nonblocking.

Nonblocking Connector

To use nonblocking connector, you have to set the protocol in server.xml

Unfortunately the result is almost the same:

Servlet no. 483 called.
Servlet no. 484 called.
Servlet no. 485 called.
Servlet no. 486 called.
May 1, 2010 5:59:24 PM org.apache.tomcat.util.net.NioEndpoint$Acceptor run
SEVERE: Socket accept failed
java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
        at java.lang.Thread.run(Thread.java:636)

Increase Open File Limit

Apparently, the blocking connector was not the issue. After some time spent with Google I have found the answer. Linux is limiting number of open connections. You can execute “ulimit -n” to see what's your limit. Luckily, it's possible to change the limit. You can either set it by “ulimit -n 32768” if you have permissions or by adding following lines to /etc/security/limits.conf (lukas is my username)

lukas            hard    nofile          32768
lukas            soft    nofile          32768

To apply this change you have to logout and login. After that, you will see this:

Servlet no. 5856 called.
Servlet no. 5857 called.
Servlet no. 5858 called.
Servlet no. 5859 called.
May 1, 2010 6:07:58 PM org.apache.tomcat.util.net.NioEndpoint$SocketProcessor run
SEVERE:
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.Arrays.copyOf(Arrays.java:2894)
	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:117)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:407)
	at java.lang.StringBuilder.append(StringBuilder.java:136)
	at java.lang.StringBuilder.append(StringBuilder.java:132)
	at java.lang.Throwable.printStackTrace(Throwable.java:529)
	at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:94)
	at java.util.logging.StreamHandler.publish(StreamHandler.java:196)
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
	at java.util.logging.Logger.log(Logger.java:476)
	at java.util.logging.Logger.doLog(Logger.java:498)
	at java.util.logging.Logger.logp(Logger.java:698)
	at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:167)
	at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:135)
	at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:755)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2080)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:636)

We have reached 5856 threads before we run out of memory. Cool.

Increase Heap Size

Let's try to increase the heap size. Just add “-Xmx2048m” to JAVA_OPTS. Before I have started running out of heap, I got to cca 11000 threads! Is it enough? I do not know, but I think it's pretty good.

Moreover, if you do a heap dump, you will see, that most of the memory is consumed by char and byte arrays. (This heap dump has been taken with approximately 5000 connections, screenshot from VisualVM)

It's understandable, we have lot of open buffers on both sides. I assume that we have at least four buffers per servlet. One for sevlet request, one for servlet response, one for URL request and one for URL response. But maybe there will be other buffers as well. To be honest, I have to admit that memory consumed by stacks would not appear here, it's probably handled by OS. But we have run out of the heap, so that's why I am talking about it.

Smaller buffers

We can try to make some of the buffers smaller. I was able to find only one setting that had some effect. Again it is connector setting in server.xml config file.

With this setting, I was able to get near to 13000 open connections.

...
Servlet no. 13327 called.
Servlet no. 13328 called.
Servlet no. 13329 called.
Servlet no. 13330 called.
Servlet no. 13331 called.
Servlet no. 13332 called.

After that the machine started to run out of physical memory, GC took ages so I had to stop the server. (Just to remind you, my test machine is two year old laptop with Intel Core 2 Duo T8100 2.1GHz with 4GB of RAM. There is 64bit Linux 2.6.32 and OpenJDK (IcedTea6 1.8) running on top of it.)

As we have seen, threads are not the major issue on modern machines. There is probably significant amout of memory consumed by the stack traces too, but I think the biggest problem are the buffers. And the important point is, that we would need the buffers even if we used asynchronous servlets! Of course, there is still some overhead connected with threads, so asynchronous libraries have their place. In fact, it would be nice to try similar experiment with asynchronous servlets. I am afraid, that I will not be able to do it, but I will be glad to help if there is some volunteer.

Please also note that your numbers may vary. After all this has been quite artificial test. I think it's simulates lot of real-life use cases, but you know, the reality is always different.

On the other hand, with more physical memory and better Tomcat configuration, we might got to higher numbers. I have heard legends about 16K threads.

I think that I will finish with my favorite message. Do not use complicated constructs unless you are sure you need them. Please remember golden rules of optimization:

The First Rule of Program Optimization: Don't do it.
The Second Rule of Program Optimization (for experts only!): Don't do it yet.

If you want to verify my results, the source code is here. If you have some comments, different results or advices, do not hesitate to add a comment.

Resources:
Tomcat connector config

Why we need asynchronous servlets

Note: If you wonder why the hell I have started to write in something that looks almost like English when apparently I do even more mistakes than in Czech, the answer is simple. I just need to practice my English (apart from that I want to be world famous, not just known in Czech Republic)

How many threads a JVM can handle?

Lukáš Křečan — Wed, 07 Apr 2010 17:02:17 +0000

Threads are expensive. They need lot of system resources, it's time-consuming to create them and JVM is able to manage only few of them. Therefore we need asynchronous processing, new libraries, new languages etc. At least, that's what I am hearing from everywhere. But is it really true? Are threads really so expensive? There is only one way how to find out. Just ask the machine.

The only thing we need to do is write a simple test. We just have to create as many sleeping or waiting threads as possible and we will see when it breaks. The reason why we are interested only in waiting threads is obvious. It has no sense to create thousands of active threads if we have only few CPUs. We want to simulate a situation, when threads are waiting for database data, incoming message or other CPU unrelated tasks.

public class CreateThreads {
	/**
	 * Thread class
	 * @author Lukas Krecan
	 *
	 */
	private static final class MyThread extends Thread
	{
		private final int number;

		public MyThread(int number) {
			this.number = number;
		}
		@Override
		public void run() {
			if (shouldPrintMessage(number))
			{
				System.out.println("Thread no. "+number+" started.");
			}
			try {
				//sleep forever
				Thread.sleep(Long.MAX_VALUE);
			} catch (InterruptedException e) {
				e.printStackTrace();
			}
		}
	}
	/**
	 * @param args
	 */
	public static void main(String[] args) {
		long startTime = System.currentTimeMillis();
		final int noOfThreads = 1000000;
		int i=0;
		try {
			for (i=0; i < noOfThreads; i++)
			{
				if (shouldPrintMessage(i))
				{
					System.out.println("Creating thread "+i+" ("+(System.currentTimeMillis()-startTime)+"ms)");
				}
				new MyThread(i).start();
			}
		} catch (Throwable e) {
			System.out.println("Error thrown when creating thread "+i);
			e.printStackTrace();
		}
	}

	private static boolean shouldPrintMessage(int i)
	{
		return i % 100 == 0;
	}
}

As you can see, there is nothing special in the test. Every hundredth threads prints a message to console and all of them sleep forever. In the main method we are creating threads and when an exception is thrown, number of created threads is printed out.

Even though there is nothing special on the test, the results are really interesting and for me quite surprising. I will let you guess the results. The test is executed on my two years old laptop with Intel Core 2 Duo T8100 2.1GHz. There is 64bit Linux 2.6.31 and OpenJDK (IcedTea6 1.6.1) running on top of it.

Try to guess how many threads are created if I do not tinker with JVM arguments at all. Is it 1K? Or 10K? Or even more? Well, it's slightly more than 32K. Usually you see result like this

Creating thread 31900 (37656ms)
Thread no. 31900 started.
Creating thread 32000 (37736ms)
Thread no. 32000 started.
Creating thread 32100 (37818ms)
Thread no. 32100 started.
Error thrown when creating thread 32172
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:614)
	at CreateThreads.main(CreateThreads.java:43)

The question is, why we can not create more threads. Are we constrained by the heap size? Or maybe stack size? No in fact, 32K is apparently Linux kernel limit. Therefore there is nothing we can do in Java, Linux is just not able to handle more threads. But I think that 32K threads is not bad at all. Please keep in mind, that you will be able to reproduce the results only on modern operating systems and JVMs. I am afraid that Windows XP results will be much worse.

Please also note, that once the threads are created and all of them sleep, the application does not consume any CPU and it consumes around 800MB of memory. I am not sure why so much memory is used, it has to be investigated.

As we have seen, in this artificial scenario threads are relatively cheap. I do not want to say, that we should not use new libraries or tools. Most of them are quite useful. The only thing I want to say is that we should think twice before prematurely optimizing our applications. Sometimes the most straightforward and easiest approach is the best.

Next time, I will try to do some more real life example with servlet container.

Test coverage – interesting only when low

Lukáš Křečan — Mon, 28 Sep 2009 19:10:59 +0000

What a nice and controversial title. But I really mean it. Level of test coverage is really interesting only when it's low. Of course, we should aim for highest coverage possible, but the higher the coverage is, the less useful information it brings. Let take the following example and let's assume there are no tests for this method.

	/**
	 Adds positive numbers. If one of the parameters is negative,
	 IllegalArgumentException is thrown.
	 * @param a
	 * @param b
	 * @return
	 */
	public int addPositive(int a, int b)
	{
		if ((a<0) || (b<0))
		{
			throw new IllegalArgumentException("Argument is negative");
		}
		return a+a;
	}

No coverage - wow, that's interesting

The fact that some piece of code is not covered at all is really interesting. It either means, that we have to start writing tests at once or that the piece of code has to be deleted. Either way we know that we should do something. Moreover, we know exactly what to do. For example, we can start with the following test.

	@Test
	public void testAddition()
	{
		assertEquals(2, math.addPositive(1, 1));
	}

It will get us partial coverage like this

Partial coverage - cool, I know what to test next

If I have partial coverage, I can see what other tests I am supposed to write. In our example I see that I have to test what happens if I pass in a negative numbers.

	@Test(expected=IllegalArgumentException.class)
	public void testNegative1()
	{
		math.addPositive(-1, 1);
	}
	@Test(expected=IllegalArgumentException.class)
	public void testNegative2()
	{
		math.addPositive(1, -1);
	}

Total coverage - I have no clue

Now we have 100% coverage. Well done! Unfortunately, I have no clue what to do next. I have no idea if I need to write more tests, I do not know if I have complete functional coverage. I have to stop looking at the coverage and I have to start thinking. You see, when the coverage was low it gave me lot of useful information, when it's high I am left on my own. For example I can think hard and come up with following complex test scenario.

	@Test
	public void testComplicatedAddition()
	{
		assertEquals(3, math.addPositive(1, 2));
	}

Please note, that the coverage was already 100% before I have written the test. So one more test could not make it higher. Nevertheless the test actually detected a bug that was there from the beginning. It means that 100% code coverage is not enough, we have to aim higher. We should care about functional coverage and not about code coverage!

That's also one of many reasons why we should do test driven development. Or at least test first development. If we write the tests before the code, we are forced to think from the beginning. In TDD we do not care about the coverage. We just think about the functionality, write corresponding tests and then add the code. Total coverage is automatically achieved.

In the opposite situation, when we write the code first and then add the tests as an afterthought, we are not forced to think. We can just write some tests, achieve the target coverage and be happy. Test coverage can be really deceitful. It can give us false sense of security. So please remember, test coverage is a good servant but a bad master.

Spring WS fault detail

Lukáš Křečan — Sat, 23 May 2009 13:59:07 +0000

Spring WS project provides nice and versatile exception handling tools. But in some scenarios predefined Exception Resolvers are not sufficient. For example if you want to provide additional error info in the soap:fault detail like in this example:


   
   
      
         SOAP-ENV:Client
         Something wrong happened
         
            Error
            BigTrouble

Fortunately it is quite easy to add similar behavior using Spring WS. You can easily extend existing SoapFaultMappingExceptionResolver and customize the fault (please note that EndpointExeption is project specific exception that provides necessary data):

 public class EndpointExceptionResolver extends SoapFaultMappingExceptionResolver {
	private static final QName CODE = new QName("code");
	private static final QName SUB_CODE = new QName("sub-code");

	@Override
	protected void customizeFault(Object endpoint, Exception ex, SoapFault fault) {
		logger.warn("Exception processed ",ex);
		if (ex instanceof EndpointException) {
			EndpointException ee = (EndpointException) ex;
			SoapFaultDetail detail = fault.addFaultDetail();
			detail.addFaultDetailElement(CODE).addText(ee.getCode());
			detail.addFaultDetailElement(SUB_CODE).addText(ee.getSubCode());
		}
	}
}

Different programming worlds

Lukáš Křečan — Sat, 11 Apr 2009 12:35:41 +0000

I have had this article in my head for almost a year but reading Practical API design book have forced me to finally write it. The idea is not new or revolutionary, on the contrary it's quite obvious: There are different programming worlds. We can use metaphor of workshops.

Jaroslav Tulach, author of Practical API design book works in one workshop. He is creating components for other programmers. Lets imagine that he is making nuts and bolts. Of course, his nuts and bolts are more sophisticated, but for sake of argument lets imagine this simple nuts and bolts scenario.

Apparently, his customers are other craftsman and their main requirement is compatibility. They want to use his parts in existing machines, they want to use the same tools they currently have. He just can't decide, that hexagonal head is not modern any more. All his customers have tools for them. Therefore, he has too maintain compatibility. Of course, in some cases he can enhance quality by using better materials, he can introduce better types of components, but once a component is sold to a customer he can't change its dimensions or other important properties. For him backward compatibility and well defined API are crucial. I think that definition for API should be: “Something that is not allowed to change”.

I am on the other hand server-side Java programmer. I write software mainly for end-users. Lets say that I am building custom-made bicycles. You know, some people have so special requirements that they can't go to a shop and just buy a bike. They need special, super-duper tailor made bike.

Usually they do not care what parts I use. They want me to meet their requirements and do it as fast as possible. In this case, I don't care about API at all. I just take existing components and put them together. Of course, I want this components to be well supported, developed in backward compatible way. But me personally, I do not care about creating APIs. I am just using them. Sure, I create my own components, but I do it only because I have to. Sometimes I have to connect two incompatible parts, sometimes there is no-one who makes the part I need. But I always prefer to use already existing parts. I do not know how in bike workshop, but in software there usually is no need to write your own library or new API. Of course, sometimes you are doing something really revolutionary, sometimes your customer requirements are really special, but in my experience, usually customers want something that has already been done hundred times before and there is already someone who makes corresponding parts.

There are only two cases when we have to define an API. The first one is, when we want to reuse custom made components in different projects. It looks like a plausible argument, but I think that in 99% of cases there already is an existing library that already does almost exactly what you need. And I think it is always better to use existing solution then reinvent a wheel.

The other situation where you have to define an API is when you interface with other supplier. For example if your customer wants to mount a rocket launcher on the bike. Then you usually have to agree on some interface between you and the rocket launcher manufacturer. I would bet that there is already a standard for this, but if there is no standard you have to define an API. In my workshop it is usually done using WebServices or other XML based format just because it is easier to change such API in almost compatible way. I usually do not use Java based API just because all the versioning and other issues it brings.

Well, it looks like I do not do anything useful. Hell, I have to defend myself. My added value is that I put those parts together. Customers do not pay me for making bicycle components they pay me for delivering a tandem-tricycles for elephants. That's my job. To build such thing as fast as possible and as cheep as possible. They do not want me to loose their money by reinventing a wheel.

So in my case creating an API is not useful. Sometimes it is even harmful. I'd like to be flexible, I want to be able to fix my mistakes as fast as possible, I'd like to change my design quite often, I do not want to maintain backward compatibility. I do not have to. The only people who will be influenced by my changes are my colleagues. It is quite easy to fix their components accordingly to my changes. At least in software world.

Of course that I try to create well encapsulated and reusable classes. Of course I try to create some kind of communication interface to all my modules. Even for internal usage, I like to have some level of cluelessness. But I would not call it API. It is something one level under API. For me it is just an implementation trick I use to make my life easier. I am not publishing it outside the team, I know that it is not stable at all. It's not an API it's just PI. Of course the distinction is not clear, but I still feel that there is a difference between a stable API and volatile internal PI. Both of them hide implementation details but only API has to be stable.

There are other types of workshops as well. There are for example mass producers. They sell bikes or software in boxes. These people live somewhere in between. They try to use existing parts as much as possible, but they have to differ somehow from their competitors. So they have to provide something special. In this case, there might be a reason to create a special component and make it standard, so they can share this component between teems working on different products. But again, I do not think that they need stable well defined API so much.

There are of course other types of workshops, but my metaphor is slowly reaching its limits so I will abandon it. What I want to say is, that there is no one size fits all approach or advice. What a surprise!

For example, you can read that you should not optimize your code because JVM will do it for you. That's true unless you work for mobile devices. I recommend you to read this, it goes against everything I believe in, yet in their use case it's valid.

There are other examples, I am sure you will come up with your own. The important thing to realize is that there are different worlds for which different approaches apply. So if you do not agree with someone, maybe the reason is that he is from completely different world.

Hibernate SQL logging

Lukáš Křečan — Sun, 05 Apr 2009 14:00:35 +0000

If you need to log SQL queries generated by Hibernate together with parameter values you do not need P6Spy. It is enough to set-up log4j or your favorite logging framework like this

log4j.logger.org.hibernate.SQL=DEBUG log4j.logger.org.hibernate.type=TRACE

Then you will see everything in your log. For example:

7820 DEBUG [org.hibernate.SQL] {main} - insert into Client (name, personalNumber, id) values (?, ?, ?) 7821 TRACE [org.hibernate.type.StringType] {main} - binding 'John Doe' to parameter: 1 7821 TRACE [org.hibernate.type.StringType] {main} - binding '123X' to parameter: 2 7821 TRACE [org.hibernate.type.LongType] {main} - binding '10' to parameter: 3

Streaming a stream

Lukáš Křečan — Fri, 27 Feb 2009 10:53:01 +0000

I have another dirty trick for you. Let's imagine that you have a document server with following sophisticated interface.

public interface DocumentServer {
	/**
	 * Stores a document.
	 * @return document id
	 */
	public String storeDocument(InputStream in);

	/**
	 * Loads document with given id.
	 * @param id
	 * @return
	 */
	public InputStream loadDocument(String id);
}

It works well, until you realize, that you want to call the DocumentServer remotely, for example using Spring HttpInvoker.

HttpInvoker is a nice small tool, that lets you make remote method invocations over HTTP without much pain. It creates dynamic proxy, serializes all attributes using Java serialization, calls remote proxy, which deserializes arguments, calls the implementation and sends the result back in the same way. It is nice, simple and it works without any problem. You can configure it in five minutes. The downside is, that you have to have Spring Java application on both ends of the wire and all arguments that you send over the wire have to be Serializable.

And that's exactly the trouble we have with our DocumentServer. There is no standard InputStream implementation that implements serializable. Lets think about it. We need to have a stream that is able to stream itself into an ObjectOutputStream. Sounds strange, but it is quite easy. We can simply write a wrapper, that implements writeObject(ObjectOutputStream) method and that takes all the bytes from the underlying stream and writes them to the ObjectOutputStream. It is simple and it does not consume much memory.

Deserialization is much harder. We have to implement readObject(ObjectInputStream in) method, load the data and store them somewhere for later use. We can store them in the memory or in a temporary file. If the streams are larger, temporary file is better choice but we have to delete them after use and the implementation gets messy.

The good news is, that we do not have to implement it, it's already done by RMIIO library. They provide SerializableInputStream class that does exactly what we need. You just have to ignore Javadoc comment saying:

An additional layer around a RemoteInputStream which makes it Serializable and an InputStream. In general, this extra layer is not necessary and I do not recommend using this class. However, in the odd case where the callee really wants to get something which is already an InputStream, this class can be useful.

The only thing we have to do is to wrap all the streams into SerializableInputStream like this

new SerializableInputStream(new DirectRemoteInputStream(inputStream));

It can be done by hand written proxy, by dynamically generated proxy or by AOP.

Of course there are other alternatives. The easiest one is to create something like ByteArayInputStream that implements Serializable. If the streams are small enough, it's the best solution.

We can also try to use Hessian, but I had some troubles when method signatures were more complicated. And of course there is still a possibility to implement own remoting mechanism. After all, HTTP is meant to transport streams. But it would require lot of boilerplate code and I do not like boilerplate code.