Set up a single Node pseudo cluster VM with vagrant and puppet

Puppet is the prefered provisioning tool for virtual clusters.

Vagrant is the best virtual machine manager for hadoop cluster development.
In the topic Introduction to Big Data c) Virtual Machine (VM), Provisioning a VM
with vagrant and puppet we saw how to build a basic VM with vagrant and puppet. N
ow we will extend our puppet syntax to create a basic puppet provision script an
d a more advanced vagrant provision script. We will need these skills to prepare
the advanced Hadoop development environment in later topics
// these scripts are current and run successfully at 15/04/2014
// download links and vagrant, puppet, operating systems and Virtualbox versions
change with // time
// if you are running these scripts later that 06/2014 then maybe some download
links and
// puppet/vagrant syntax may need fine tuning
1. create a directory like /home/<user>/project
1.2 set an env var VAGRANT_HOME=/home/<user>/project to this directory
1.3 in the project directory mkdir manifests
1.4 put the file below base-hadoop.pp in this directory
2. in the project directory type vagrant init
2.1 replace the contents of the auto generated Vagrantfile with the Vagrantfile belo
w
2.3 put the provision.sh file in the project directory
3. In the project directory type vagrant up to create the box
Screenshot from 2014-04-12 19:15:46.png

this is the puppet provisioning phase

Screenshot from 2014-04-12 19:17:31.png

this is the post installation provision.sh shell script phase, puppet provisioni
ong has completed successfully
Screenshot from 2014-04-12 19:19:24.png
do not worry about the red text this is normal, errors while in red are flagged
with ERROR
4. type vagrant ssh to enter the box
Screenshot from 2014-04-12 19:26:37.png

4.1 type source /etc/environment to refresh the configured environment variables

Screenshot from 2014-04-12 19:28:28.png
4.2 type hadoop fs -ls / to check that hadoop is configured and able to run
Screenshot from 2014-04-12 19:38:22.png
4.3 type exit to leave the running box
4.3.1 type vagrant halt to shut down the box
4.3.2 type vagrant destroy to remove the box and recover the disk space if needed
Vagrant and Puppet provisioning files
Puppet provisioning file

base-hadoop.pp put in VAGRANT_HOME/manifests

group { "puppet":
ensure => "present",
}
Exec { path => [ "/bin/", "/sbin/" , "/usr/bin/", "/usr/sbin/" ] }
$hadoop_home = "/home/vagrant"
exec {

apt-get update :
command => apt-get update ,

}
package { "openjdk-6-jdk" :
ensure => present,
require => Exec[ apt-get update ],
}

exec { "download_hadoop":
command => "wget -O /home/vagrant/hadoop-1.2.1.tar.gz http://apache.mirrors.timp
orter.net/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz",
path => $path,
unless => "ls /home/vagrant | grep hadoop-1.2.1",
require => Package["openjdk-6-jdk"],
}
exec { "unpack_hadoop" :
command => "tar -zxf /home/vagrant/hadoop-1.2.1.tar.gz",
path => $path,
creates => "${hadoop_home}/hadoop-1.2.1",
require => Exec["download_hadoop"],
}

Vagrantfile put in VAGRANT_HOME

VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "ubu-hadoop"
config.vm.box_url = "http://files.vagrantup.com/lucid64.box"
config.vm.synced_folder "./", "/vagrant", id: "vagrant-root"
config.vm.provision :puppet do |puppet|
puppet.manifests_path = "manifests"
puppet.options = [ --verbose ]
puppet.manifest_file = "base-hadoop.pp"
end
config.vm.define :setVars do |setVars|
setVars.vm.provision :shell, :path => "provision.sh"
end
config.vm.provision "shell", inline: "source /etc/environment"
end

shell script provision.sh put in VAGRANT_HOME

wget ftp://mirror.reverse.net/pub/apache/maven/maven-3/3.1.1/binaries/apache-mav
en-3.1.1-bin.tar.gz

# set stuff up
tar -xvf apache-maven-3.1.1-bin.tar.gz
mv apache-maven-3.1.1 maven
# create final env settings centos
sudo sh -c "cat >> /etc/environment" << EOF
export HADOOP_HOME=/home/vagrant/hadoop-1.2.1
export MAVEN_HOME=/home/vagrant/maven
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/jre
PATH=$PATH:$JAVA_HOME/bin:$MAVEN_HOME/bin:$HADOOP_HOME/bin
export PATH
EOF
# set env var s
source /etc/environment
#clean up
sudo rm *.tar.*