Testing your Tests with Cobertura

You can use Cobetura to improve any project, even ones that already use Clover. 

Complex java applications involve components that are similar, yet loaded dynamically.  For example:  In a typical OSGi style application, we might have many plugins added at runtime that implement a certain behaviour or provide a service.  The quintessential example of this are the hadoop filesystem implementations, which are loaded from the classpath, defined in an XML file.

One of the goals of the HCFS initiative is to evaluate how much test coverage there is for the FileSystem API, which is implemented in two ways in the hadoop code base: The RawLocalFileSystem and the DistributedFileSystem.  You can track the work we're doing here  https://wiki.apache.org/hadoop/HCFS/Progress.

However, both of these implementations are never (or rarely) called directly in code: Rather, they are represented to the test libraries as simply FileSystem implementations.

This means there's very low static test coverage of File Systems in hadoop.

So, how can we REALLY evaluate the test coverage?  We need to monitor method calls at runtime.


Enter Cobetura.

Cobetura is a test library which brings aspect oriented style injection of byte code into a single maven plugin.  Here's how to we've set it up to evaluate FileSystem coverage.

And add Cobetura 

Note: For mac users, you'll have to do some tricks to get classes.jar and tools.jar resolved, because corbertura explicitly references tools.jar.  See http://permalink.gmane.org/gmane.comp.java.maven-plugins.mojo.user/4320 for details.  My really weird way of doing this was to do "sudo cp /Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/Home/lib/* /Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/Home/Classes/".  It basically dumps all the jar files into the Classes directory which is mentioned in the RuntimeException thrown by Cobetura.

   <build>
     <pluginManagement>
       <plugins>
-        <plugin>
+           <plugin>
+               <groupId>org.codehaus.mojo</groupId>
+               <artifactId>cobertura-maven-plugin</artifactId>
+               <version>2.6</version>
+           </plugin>
+             <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-dependency-plugin</artifactId>
           <version>2.4</version>

Run the maven goal: 

mvn cobetura:cobetura 

And then you can open up ./hadoop-common-project/hadoop-common/target/site/cobertura/index.html  in your browser.   You should see something like the image at the top of this post.  This is the "real" test coverage you have.


Now what I really want to know is : how can I find out where the test coverage is coming from?  After all, since the coverage reported by cobetura is not static, there isnt any direct way to trace the coverage back to individual tests.  If I figure out a way I'll update this post.