Hadoop 2.7.3 et Hbase1.2.2on Ubuntu 16.

04

Le Hadoop nécessite Java pour s'exécuter. Tout d'abord, nous allons installer Java et nous
installerons Hadoop.mais avant nous allons mettre à jour notre liste de paquets:

Étape 1 - Mise à jour:

kacem@bigdata:~$ sudo apt-get update

Étape 2 - Installation de Java :

kacem@bigdata:~$ sudo apt-get install default-jdk

Étape 3 - Créer un groupe:

Nous allons créer un groupe, configurer les autorisations du groupe sudo,
puis ajouter l'utilisateur au groupe. Ici 'hadoop' est un nom de groupe et
'manhadoop' est un utilisateur du groupe.

kacem@bigdata:~$ sudo addgroup hadoop

kacem@bigdata:~$ sudo adduser --ingroup hadoop manhadoop
Adding user `manhadoop' ...
Adding new user `hduser' (1001) with group `hadoop' ...
Creating home directory `/home/hduser' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for hduser
Enter the new value, or press ENTER for the default
Full Name []:
Room Number []:
Work Phone []:
Home Phone []:
Other []:
Is the information correct? [Y/n] Y

kacem@bigdata:~$ groups manhadoop

manhadoop : hadoop

Ajoutez le 'manhadoop' à la liste sudoers pour que manhadoop puisse faire

des tâches d'administration.

kacem@bigdata:~$ sudo visudo

Ajouter une ligne sous:
##Allow member of group sudo to execute any command anywhere in the
format. (Right click and Paste)
manhadoop ALL=(ALL)ALL

Appuyez sur ctrl + x, Y enter entrer

Déconnectez-vous de votre système et reconnectez-vous en tant que

manhadoop.
Étape 4 - Installation du serveur open-ssh:

manhadoop@bigdata:~$ sudo apt-get install ssh

manhadoop@bigdata:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/manhadoop/.ssh/id_rsa):
Created directory '/home/manhadoop/.ssh'.
Your identification has been saved in /home/manhadoop/.ssh/id_rsa.
Your public key has been saved in /home/manhadoop/.ssh/id_rsa.pub.
The key fingerprint is:
50:6b:f3:fc:0f:32:bf:30:79:c2:41:71:26:cc:7d:e3 manhadoop@bigdata
The key's randomart image is:
+--[ RSA 2048]----+
| .oo.o |
| . .o=. o |
| . + . o . |
| o = E |
| S + |
| . + |
| O + |
| O o |
| o.. |
+-----------------+

manhadoop@bigdata:~$ cat $HOME/.ssh/id_rsa.pub >>

$HOME/.ssh/authorized_keys

manhadoop@bigdata:~$ ssh localhost

The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is
SHA256:e8SM2INFNu8NhXKzdX9bOyKIKbMoUSK4dXKonloN8JY.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-47-generic x86_64)
...

Hadoop Installation
Téléchargez Hadoop
j'utilise hadoop-2.7.3.tar.gz, mais cela devrait fonctionner avec
n'importe quelle autre version.

Étape5 :Téléchargez hadoop-2.7.3.tar.gz et enregistrez-le dans manhadoop / Bureau.

Étape 6 - Création du répertoire hadoop:
manhadoop@bigdata:~$ sudo mkdir -p /usr/local/hadoop

Étape 7 Changez la propriété et les permissions du répertoire / usr / local / hadoop. Ici,
'manhadoop' est un nom d'utilisateur Ubuntu.
manhadoop@bigdata:~$ sudo chown -R manhadoop:hadoop /usr/local/hadoop
manhadoop@bigdata:~$ sudo chmod -R 777 /usr/local/hadoop
Étape 8 Décompressez le fichier hadoop-2.7.3.tar.gz:
manhadoop@bigdata:~/Bureau$ tar xvzf hadoop-2.6.0.tar.gz
Étape 9 Déplacez le contenu du dossier hadoop-2.7.3 dans / usr / local / hadoop:

manhadoop@bigdata:~/Bureau$ sudo mv hadoop-2.7.3 /* /usr/local/hadoop

Étape 10 configuration les fichier hadoop
10-1 Editez le fichier $ HOME / .bashrc en ajoutant le chemin java et hadoop:
Avant d'éditer le fichier .bashrc dans le répertoire personnel de
manhadoop, nous devons trouver le chemin où Java a été installé pour
définir la variable d'environnement JAVA_HOME en utilisant ce qui suit
manhadoop@bigdata:~$ update-alternatives --config java
manhadoop@bigdata:~$ gedit ~/.bashrc
on ajoute les lines suivantes dans le fichier bashrc:
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_INSTALL=/usr/local/hadoop
export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
export HADOOP_OPTS="$HADOOP_OPTS
-Djava.library.path=$HADOOP_INSTALL/lib/native"
export JAVA_LIBRARY_PATH=$HADOOP_INSTALL/lib/native
export LD_LIBRARY_PATH=$HADOOP_INSTALL/lib/native:$LD_LIBRARY_PATH
export YARN_EXAMPLES=$HADOOP_INSTALL/share/hadoop/mapreduce
export HADOOP_MAPRED_STOP_TIMEOUT=30
export YARN_STOP_TIMEOUT=30
manhadoop@bigdata:~$ source ~/.bashrc
10-2 Nous devons définir JAVA_HOME en modifiant le fichier hadoop-env.sh:

manhadoop@bigdata:~$ gedit /usr/local/hadoop/etc/hadoop/hadoop-env.sh

export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
export HADOOP_HOME_WARN_SUPPRESS="TRUE"
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

10-3/usr/local/hadoop/etc/hadoop/core-site.xml:

Le fichier /usr/local/hadoop/etc/hadoop/core-site.xml contient les propriétés de configuration

utilisées par Hadoop lors du démarrage.Ce fichier peut être utilisé pour remplacer les paramètres par
défaut avec lesquels Hadoop commence.

manhadoop@bigdata:~$ sudo mkdir -p /app/hadoop/tmp

manhadoop@bigdata:~$ sudo chown -R manhadoop:hadoop /app/hadoop/tmp
manhadoop@bigdata:~$ sudo chmod -R 777 /app/hadoop/tmp
manhadoop@bigdata:~$ gedit /usr/local/hadoop/etc/hadoop/core-site.xml

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>

<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>
</configuration>
 1-4 /usr/local/hadoop/etc/hadoop/mapred-site.xml:

manhadoop@bigdata:~$ cp /usr/local/hadoop/etc/hadoop/mapred-
site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml
manhadoop@bigdata:~$ gedit /usr/local/hadoop/etc/hadoop/mapred-site.xml

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

</configuration>
10-5 /usr/local/hadoop/etc/hadoop/yarn-site.xml:
manhadoop@bigdata:~$ gedit /usr/local/hadoop/etc/hadoop/yarn-site.xml

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
10-6 /usr/local/hadoop/etc/hadoop/hdfs-site.xml:

Avant d'éditer ce fichier, nous devons créer deux répertoires qui

contiendront le namenode et le datanode pour cette installation Hadoop.

manhadoop@bigdata:~$ sudo mkdir -p /usr/local/hadoop_store/hdfs/namenode

manhadoop@bigdata:~$ sudo mkdir -p /usr/local/hadoop_store/hdfs/datanode
manhadoop@bigdata:~$ sudo chown -R manhadoop:hadoop /usr/local/hadoop_store
manhadoop@bigdata:~$ sudo chmod -R 777 /usr/local/hadoop_store
manhadoop@bigdata:~$ gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is
created.
The default is used if replication is not specified in create time.
</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop_store/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop_store/hdfs/datanode</value>
</property>
</configuration>
manhadoop@bigdata:~$ hadoop namenode -format

hduser@laptop:~$ hadoop namenode -format

DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

18/02/11 10:01:33 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = laptop/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.7.3
...
...
...
18/02/11 10:01:33 INFO util.ExitUtil: Exiting with status 0
18/02/11 10:01:33 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at laptop/127.0.1.1
************************************************************/

manhadoop@bigdata: /usr/local/hadoop/sbin$ ls
distribute-exclude.sh start-all.cmd stop-balancer.sh
hadoop-daemon.sh start-all.sh stop-dfs.cmd
hadoop-daemons.sh start-balancer.sh stop-dfs.sh
hdfs-config.cmd start-dfs.cmd stop-secure-dns.sh
hdfs-config.sh start-dfs.sh stop-yarn.cmd
httpfs.sh start-secure-dns.sh stop-yarn.sh
kms.sh start-yarn.cmd yarn-daemon.sh
mr-jobhistory-daemon.sh start-yarn.sh yarn-daemons.sh
refresh-namenodes.sh stop-all.cmd
slaves.sh stop-all.sh
manhadoop@bigdata:/usr/local/hadoop/sbin $ start-dfs.sh
manhadoop@bigdata: /usr/local/hadoop/sbin$ start-yarn.sh
manhadoop@bigdata: /usr/local/hadoop/sbin$ jps

2248 DataNode
2682 ResourceManager
2416 SecondaryNameNode
2128 NameNode
2800 NodeManager
26106 Jps
manhadoop@bigdata:/usr/local/hadoop/sbin $ stop-dfs.sh
manhadoop@bigdata: /usr/local/hadoop/sbin$ stop-yarn.sh
http://localhost:50070/
Hbase installation:
Étape1 :Téléchargez hbase-1.2.2.tar.gz et enregistrez-le dans manhadoop / Bureau.

Étape 2 - Création du répertoire hbase:

manhadoop@bigdata:~$ sudo mkdir -p /usr/local/hbase

Étape 3 Changez la propriété et les permissions du répertoire / usr / local / hbase. Ici, 'manhadoop'
est un nom d'utilisateur Ubuntu.
manhadoop@bigdata:~$ sudo chown -R manhadoop:hadoop /usr/local/hbase
manhadoop@bigdata:~$ sudo chmod -R 777 /usr/local/hbase
Étape 4 Décompressez le fichier hbase-1.2.2.tar.gz:
source ~/.bashrc manhadoop@bigdata:~/Bureau$ tar xvzf hadoop-1.2.2.tar.gz
Étape 5 Déplacez le contenu du dossier hbase-1.4.2 dans / usr / local / hbase:

manhadoop@bigdata:~/Bureau$ sudo mv hbase-1.2.2 /* /usr/local/hbase

Étape 6 configuration les fichier hadoop

6-1 Editez le fichier $ HOME / .bashrc en ajoutant le chemin java et hbase:
on ajoute les lines suivantes dans le fichier bashrc:
manhadoop@bigdata:~$ gedit ~/.bashrc
export HBASE_HOME=/usr/local/hbase
export PATH=$PATH:$HBASE_HOME/bin
manhadoop@bigdata:~$ source ~/.bashrc

6-2 Nous devons définir JAVA_HOME en modifiant le fichier hbase-env.sh:

manhadoop@bigdata:~$ gedit /usr/local/hbase/conf/hbase-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HBASE_REGIONSERVERS=${HBASE_HOME}/conf/regionservers
export HBASE_MANAGES_ZK=true
6-3 /usr/local/hbase/conf/hbase-site.xml:

manhadoop@bigdata:~$ gedit /usr/local/hbase/conf/hbase-site.xml

<configuration>

<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:54310/hbase</value>
</property>

<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>

<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2222</value>
</property>

<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/manhadoop/zookeeper</value>
</property>

</configuration>

manhadoop@bigdata: /usr/local/hadoop/sbin$ jps

2248 DataNode
2682 ResourceManager
2416 SecondaryNameNode
2128 NameNode
2800 NodeManager
26106 Jps
manhadoop@bigdata: /usr/local/hbase/bin$ start-hbase.sh
localhost: starting zookeeper, logging to
/usr/local/hbase/bin/../logs/hbase-manhadoop-zookeeper-laptop.out
starting master, logging to /usr/local/hbase/logs/hbase-manhadoop-master-
laptop.out
starting regionserver, logging to /usr/local/hbase/logs/hbase-manhadoop-1-
regionserver-laptop.out

manhadoop@bigdata: /usr/local/hbase/bin$ jps

2274 DataNode
2158 NameNode
2769 NodeManager
3310 HQuorumPeer
3490 HRegionServer
2439 SecondaryNameNode
3373 HMaster
3615 Jps
2650 ResourceManager

manhadoop@bigdata: /usr/local/hbase/bin$ hbase shell

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hbase/lib/slf4j-log4j12-
1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-
1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.2.2, r67592f3d062743907f8c5ae00dbbe1ae4f69e5af, Tue Oct 25
18:10:20 CDT 2016

hbase(main):001:0>
http://localhost:16010