Configuring Tachyon against Apache BigTop

We recently added tachyon 0.5 to apache bigtop.  This was based off of the work done in the fedora community by tim st claire and huamin chen.

NOTE: I'm currently updating the tachyon puppet recipes in bigtop to automate ALL of this ! But in the interim, to get it working, a couple of modifications you need to do.

Right now, its not published, so you have to build it , using:

gradle tachyon-yum

Then, you can reference that yum repository from vagrant, by adding a couple of lines to the vagrant files we have.  This will allow yum to install anything w/o checksums, and also, it will add the output/ directory (which is what is created when you run commands from the top level of bigtop), into the VM that vagrant creates. This way, you can augment the standard bigtop release with custom RPMs you may be building on the fly.

Then, you can reference that yum repository from vagrant, by adding a couple of lines to the vagrant files we have.

1) First, make sure you add the tachyon impl to hdfs.

2) Then, make sure to copy the tachyon-client jar (from /usr/lib/tachyon/) into /usr/lib/hadoop/lib/, so that you don't need to add it to -libjars everytime you call tacyhon.


3) Also, update tachyon-env.sh to point to your HCFS under file system (in bigtop, default is HDFS).  So you add lines such as the hdfs uri below.

4) Then, make sure to copy the tachyon-client jar (from /usr/lib/tachyon/) into /usr/lib/hadoop/lib/, so that you don't need to add it to -libjars everytime you call tacyhon.

5) Now, you can use tachyon:// as the URLfrom the hadoop fs shell, or from yarn jobs.