Today I have a detective story for you. Let’s start with basic information. Victim is well known Tomcat 5.5 server. He lives in a virtual server, with 8 processors and 256 megabytes of memory. There are also several servants living in the house. First of all, there is Debian who does basically all the housekeeping. Then there is native Apache servant who communicates with the outside world. In the cellar lives MySql daemon who keeps all important data.

The main occupation of the Tomcat server is to run my great mvnindex.org project. It is quite easy job. First of all, no one except me is using it. Secondly, the main task of the server is calling the database and returning the results. And of course once per day he has to index the repository.

Even thou his task is simple, sometimes I find Tomcat lying dead. Sometimes he manages to write his last words to his log.

Java HotSpot(TM) Client VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated

Sometimes he dies without any whisper. At first glance it looks like a natural death caused by lack of heap memory. But there is some circumstantial evidence against this theory. First of all, this mystical sudden death usually happens when there is some visit in the house. For example when I am building new version of the application. Imagine that. I am running mvn clean install command and Tomcat, who is running in a separate process, in separate JVM (1.6.0-b105) under another user suddenly dies. That is strange.

Moreover, adding heap memory using java -Xmx parameter does not help. Sometimes Tomcat lives for several days, sometimes he dies twice per day.

Now you have all the data you need to solve the mystery. Is it a murder? Is it a suicide? Is there a memory leak in the application? Is it a sabotage?

I will not tell you the solution right know. I am sure that someone will solve it in the discussion below. If not, I will publish the solution in few days.