Hacking around with SBT: Fat Spark local executable Jars and MergeStrategies

SBT is a great tool for building scala projects :  The build system (similar to gradle) allows you to write code to define your build steps, rather than imposing a restrictive declarative syntax.  However.  I ran into some issues trying to build a spark app which bundled a lot of jars, in particular, the structure, layout, and plugins associated with non-trivial SBT projects seemed a little tricky to find.

• For example, unlike other build tools, SBT has multiple files / folders (there is a project/ directory) which define the build environment.  
• Likewise, in order to do some common important tasks (like dependency investigation), you need to add plugins to some of those files - and later, in your build file, you will use those plugins.   
• Finally, SBT needs a lot of memory, and its not immediately clear off the bat how to set this up (seems to vary between platforms).

In any case, here are some of the things I had to do when hacking together a single integrated build for a Spark application which I wanted to be able to execute locally as a raw jar file with bundled dependencies. 

1) Two awesome plugins that you should have if doing anything non-trivial in SBT

This is really obvious once you read the docs and see a few projects.  There is a "project" directory, and under that, you can add a plugins.sbt file.    There are a couple of plugins that I think I'll probably include in almost every project from now on.

• https://github.com/jrudolph/sbt-dependency-graph
• https://github.com/sbt/sbt-assembly

For those getting started, in sbt 0.13.6 my plugins file looks like this.

Adding sbt-dependency-graph and sbt-assembly.

2) Dependency graphs and MergeStrategies.

Now that you can build non-trivial jars, you will need to understand MergeStrategies.   When building a single jar file, certain files might overlap (i.e application.conf or reference.conf for akka/spark apps).  You can work around this by making a MergeStrategy code block in your build.sbt file, which tells sbt assembly tasks how to merge files in.  You can choose "first", "last", "concat" and so on.  In my case, I used "concat" to fix this issue building spark apps.

Relating to the MergeStrategy, you might want to investigate conflicting jars to see when certain files conflict.  The easy way to do this is to (1) launch sbt and (2) run dependency-graph.  This gives you a graph quite similar to "mvm dependency:tree" - a picture of all jars brought in, including transitively, so you can do manual exclusions wherever you need to.  Sometimes a manual exclusion is a better idea (or more feasible) than doing a concat or pick first / pick last strategy.  For obvious reasons.

3) SBT NEEDS MEMORY !

If you look online, there are lots of ways for adding memory to SBT.  I installed SBT using brew for my mac, in which case the SBT_OPTS variable wasn't being used for some reason.  So, a simple way to  add memory for builds (this is critical for building fat jars with lots of dependencies), is to directly edit the SBT configuration files. 

For me this means editing /usr/local/etc/sbtopts.  To add, for example 2G of memory as the default JVM size for SBT launching, your sbtopts file will look like this:

Add mem 2000 = 2G JVM heap for SBT builds.