 |
| more virtual big data love w/ gluster, vagrant, and mattf's little fake disk hack :) |
For those of you who need to spin up virtual gluster clusters for development and testing:
Just finished creating a vagrantized, fully automated, totally rebuildable and teardownable two node fedora 19 gluster setup and shared it on the forge!
It uses fedora19 but since its all vagrant powered, you dont need to grab or download a distro or iso or anything, just clone the git repo, run vagrant up, and let vagrant automagically pull down and manage your base box and set up the rest for you.
clone it here: https://forge.gluster.org/vagrantSo what does this do? This basically means that you can spin up 2 vms , from scratch, by installing vagrant and then, literally, typing:
git clone git@forge.gluster.org:vagrant/fedora19-gluster.git
cd fedora19-gluster
ln -l Vagrantfile_cluster Vagrantfile
vagrant up --provision
Does it work? Yes ! After watching it spin up , you can ssh in to the cluster by typing:
And destroy the same two node cluster by running:vagrant ssh gluster1
vagrant destroyHow it works
To grok the basics of how it works:
- First: Check out the Vagrantfile. That file defines some of the basic requirements, in particular, the static ips that the peers have.
- How does the cluster install get coordinated? Now, look at the provisioning scripts referenced from the vagrant file. Those run after the basic box is set up. There is a little bit of hacking to ensure that, when the final box is setup, only then is the peer probing done... but otherwise its pretty straight forward (actually that could be dehacked by simply having a second provision script in the 2nd node of the Vagrantfile... but I only just learned on vagrant irc that you could have multiple provisioners.
- What about installing gluster? Thankfully, Fedora now has all the gluster rpms as supported for standard F19. So it was super easy to yum install them.
- Finally: the bricks: And you will also see that there is a special methodology for creating gluster "bricks" on each machine. That simple little trick for setting up a fake disk (using the truncate) command [+1 to spinningmatt.wordpress.com at redhat for showing me this]!
.............................. Example .........................................
#Below : ssh into the first node, create a file, and ls it in the other node to confirm that the cluster is working.
[jays-macbook]$ vagrant ssh gluster1
[vagrant@gluster1 ~]$ sudo touch /mnt/glusterfs/a
[vagrant@gluster1 ~]$ ssh 10.10.10.12
[vagrant@gluster2 ~]$ ls /mnt/glusterfs/
[vagrant@gluster1 ~]$ sudo touch /mnt/glusterfs/a
[vagrant@gluster1 ~]$ ssh 10.10.10.12
[vagrant@gluster2 ~]$ ls /mnt/glusterfs/
Help wanted to maintain this and make development clusters for different
gluster use cases.
gluster use cases.
- bug reporting
- testing functionality of different tuning/config options
- testing hadoop interop
Lots of cool stuff we can do with this. right now im attempting to bridge it with a vagrantized psuedodistributed hbase implementation here: https://github.com/jayunit100/vagrant-hbase . Stay tuneeddddddddd
Err, this seems to lack the description on how to set up the disks
ReplyDeleteRegarding the disks... (1) I can point you at a script which makes a "fake" disk by using the "truncate" command: https://forge.gluster.org/vagrant/fedora19-gluster/blobs/master/vagrant-gluster-examples/data/util/singlenodesetup.sh . But the better way to see how it works is, first run "vagrant up" - that will do it all for you. After that, trace the code (From the Vagrantfile, to the provisioner scripts), and you will see how it all works. For example, the singlenodesetup.sh script is ultimately going to be called by the Vagrant provisioning process.
ReplyDeleteThanks for the response - looking into the provisioning scripts I see what's intended, however this isn't being run in my case. Caveat: I've only recently starting using Vagrant so the error might be mine. I'm seeing the following error:
ReplyDelete[gluster1] Configuring and enabling network interfaces...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!
ARPCHECK=no /sbin/ifup eth1 2> /dev/null
Stdout from the command:
ERROR : [/etc/sysconfig/network-scripts/ifup-eth] Device eth1 does not seem to be present, delaying initialization.
and the provisioning script does not appear to be run. Note that ethN names no longer exist in Fedora so the ifup command makes no sense. Note that I'm using Vagrant 1.4.2 and VirtualBox 4.3.6
Hmm, i was using vagrant 1.3.3 ive added your note to https://github.com/mitchellh/vagrant/issues/2614#issuecomment-32188507 .... can you follow up with more details ? Thanks for the update. let me know if it works on 1.3.3 for you ?
ReplyDeleteI had similar problems with vagrant 1.3.3 (and VirtualBox 4.0). I found that reconfiguring the VMs to use paravirtualised networking (by default VirtualBox was using emulated Intel NICs) the interfaces were named eth0/eth1 and now the networking error has disappeared. After re-starting the machines, the provisioning scripts are being run. I'll test again with more recent Vagrant and VirtualBox. Out of interest, when you provision a VM, what type of NICs are created? Vagrant docs suggest that nic_type: "virtio" should do the right thing, but your Vagrant file seems to only define one i/f.
ReplyDeleteI thought it was working: the provisioning scripts are being run but the return frequent errors "sudo: sorry, you must have a tty to run sudo" so the peer probe isn't done, the 'thin' bricks are not created and volumes are not created. This is the two-node cluster btw.
ReplyDeleteThanks for this daniel. Its funy, i seem to recall both the TTY Error and the "eth1 not found" errors. Just a hunch, but , did you change the vagrant box are you using ? I'm using this one : http://rhbd.s3.amazonaws.com/vagrant/fedora19B.box . Let me know if that makes your life any better ....?
ReplyDeleteThat is the box I am using. The vagrant file specifies that same box using the s3:// schema in the URI, which causes curl to throw an error.
ReplyDeletehey i just saw this. still having this issue ? I can grab it from http://rhbd.s3.amazonaws.com/vagrant/fedora19B.box immediately in firefox.
DeleteFYI I dont even know if this works anymore, its been like 5 months which is a decade in Fedora years :)
ReplyDelete