Test coverage – interesting only when low

What a nice and controversial title. But I really mean it. Level of test coverage is really interesting only when it's low. Of course, we should aim for highest coverage possible, but the higher the coverage is, the less useful information it brings. Let take the following example and let's assume there are no tests for this method.

	/**
	 Adds positive numbers. If one of the parameters is negative,
	 IllegalArgumentException is thrown.
	 * @param a
	 * @param b
	 * @return
	 */
	public int addPositive(int a, int b)
	{
		if ((a<0) || (b<0))
		{
			throw new IllegalArgumentException("Argument is negative");
		}
		return a+a;
	}

No coverage - wow, that's interesting

The fact that some piece of code is not covered at all is really interesting. It either means, that we have to start writing tests at once or that the piece of code has to be deleted. Either way we know that we should do something. Moreover, we know exactly what to do. For example, we can start with the following test.

	@Test
	public void testAddition()
	{
		assertEquals(2, math.addPositive(1, 1));
	}

It will get us partial coverage like this
Partial coverage

Partial coverage - cool, I know what to test next

If I have partial coverage, I can see what other tests I am supposed to write. In our example I see that I have to test what happens if I pass in a negative numbers.

	@Test(expected=IllegalArgumentException.class)
	public void testNegative1()
	{
		math.addPositive(-1, 1);
	}
	@Test(expected=IllegalArgumentException.class)
	public void testNegative2()
	{
		math.addPositive(1, -1);
	}

Total coverage - I have no clue

Now we have 100% coverage. Well done! Unfortunately, I have no clue what to do next. I have no idea if I need to write more tests, I do not know if I have complete functional coverage. I have to stop looking at the coverage and I have to start thinking. You see, when the coverage was low it gave me lot of useful information, when it's high I am left on my own. For example I can think hard and come up with following complex test scenario.

	@Test
	public void testComplicatedAddition()
	{
		assertEquals(3, math.addPositive(1, 2));
	}

Please note, that the coverage was already 100% before I have written the test. So one more test could not make it higher. Nevertheless the test actually detected a bug that was there from the beginning. It means that 100% code coverage is not enough, we have to aim higher. We should care about functional coverage and not about code coverage!

That's also one of many reasons why we should do test driven development. Or at least test first development. If we write the tests before the code, we are forced to think from the beginning. In TDD we do not care about the coverage. We just think about the functionality, write corresponding tests and then add the code. Total coverage is automatically achieved.

In the opposite situation, when we write the code first and then add the tests as an afterthought, we are not forced to think. We can just write some tests, achieve the target coverage and be happy. Test coverage can be really deceitful. It can give us false sense of security. So please remember, test coverage is a good servant but a bad master.

Tags:

2 Responses to “Test coverage – interesting only when low”

  1. lzap Says:

    Yes. Nice.

    And always remember 100% CC does not mean you have covered 100% of all possible program states.

  2. Adam Hošek Says:

    Writing test after the actual code is just like the necessary evil to satisfy ourselves or our superiors. I don't say it's better to do not write test at all. All in all I agree with Lukáš.

    I've posted this article to DZone.com by the way 🙂 (http://dzone.com/yJtU)