Using SCoverage with on Apache Spark - or any scala project.

SCoverage is a tool which instruments test coverage for Scala.  It currently supports scala 2.11 and up, but it works with 2.10 if you want it to :).  The maven Scala build for spark actually employs SBT under the hood.   Heres a quick overview of getting it working. 

1) Turning code coverage on and off.

Since SCoverage only supports 2.11 (officially) there are chances that certain scala features might break it.  Also, in general, there are areas of your code you may decide you don't want to instrument for testing.   To do this, you can use the $COVERAGE-OFF directive, which tells SCoverage to ignore bits of your code.

2) Now with coverage turned on/off, we can configure our maven pom file to run sCoverage with the tests. We do this by putting in the directivesto the pom file which controls the scala parts of our source code (i.e. in spark, we do this in core/pom.xml).

In this case, I added a REGEX system property variable which does package level filtering.  Its not really required but you can invoke it like so ( -DREGEX=".*network.*" -Pcoverage clean test ). 

3) Finally, when we run our tests, we can launch them with the coverage profile.  This tells maven to do code AST instrumentation as part of the tests.  The instrumentation then is used by SCoverage to analyze what parts of our scala code are covered by our unit tests.

mvn -Pcoverage clean test 

4) We can confirm that our tests are indeed being covered if we look here

We can see the scoverage logs, regarding how many classes were, and were not covered.

5) When done, scoverage gives us a lovely html report ! You can serve it up using something like python's simple web server.  Just copy it into a tarball and launch it in python. 
    

    cd scoverage-report/
    python -m SimpleHTTPServer 8000 

6) And there you go ! Now you should have your coverage reports served up to the web.  They will look something like this.

And thats it ! Looking forward to a follow up post from dr will  http://chapeau.freevariable.com/    !