hadoop 安装
http://hadoop.apache.org/docs/r2.6.5/hadoop-project-dist/hadoop-common/ClusterSetup.html
1. 环境¶
| 序列号 | IP地址 | 主机名 | 角色 | 安装软件 |
|---|---|---|---|---|
| 1 | 192.168.186.10 | master | hadoop master |
java1.8，hadoop2.6.5 |
| 2 | 192.168.186.11 | slave1 | hadoop slave1 |
java1.8，hadoop2.6.5 |
| 3 | 192.168.186.12 | slave2 | hadoop slave2 |
java1.8，hadoop2.6.5 |
2. 优化¶
参考我的博客 centos7 详细优化 http://mail.loocha.com:10000/linux/tutorial/linux.optimization/
2.1 修改主机名和解析¶
快速操作
echo 'master' > /etc/hostname hostname master bash cat>>/etc/hosts<<EOF 192.168.186.10 master 192.168.186.11 slave1 192.168.186.12 slave2 EOF ping -c1 master ping -c1 slave1 ping -c1 slave2
详细操作
[root@localhost ~]# echo 'master' > /etc/hostname [root@localhost ~]# hostname master [root@localhost ~]# bash [root@master ~]# [root@master ~]# cat>>/etc/hosts<<EOF > 192.168.186.10 master > 192.168.186.11 slave1 > 192.168.186.12 slave2 > EOF [root@master ~]# ping -c1 master PING master (192.168.186.10) 56(84) bytes of data. 64 bytes from master (192.168.186.10): icmp_seq=1 ttl=64 time=0.105 ms --- master ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.105/0.105/0.105/0.000 ms [root@master ~]# ping -c1 slave1 PING slave1 (192.168.186.11) 56(84) bytes of data. 64 bytes from slave1 (192.168.186.11): icmp_seq=1 ttl=64 time=0.191 ms --- slave1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.191/0.191/0.191/0.000 ms [root@master ~]# ping -c1 slave2 PING slave2 (192.168.186.12) 56(84) bytes of data. 64 bytes from slave2 (192.168.186.12): icmp_seq=1 ttl=64 time=0.480 ms --- slave2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.480/0.480/0.480/0.000 ms
另外两台类似，省略
2.2 关闭selinux/防火墙¶
sed -i 's@SELINUX=enforcing@SELINUX=disabled@g' /etc/selinux/config setenforce 0 systemctl disable firewalld # 开机不自启防火墙 systemctl stop firewalld # 关闭防火墙
[root@master ~]# sed -i 's@SELINUX=enforcing@SELINUX=disabled@g' /etc/selinux/config [root@master ~]# setenforce 0 setenforce: SELinux is disabled [root@master ~]# systemctl disable firewalld # 开机不自启防火墙 [root@master ~]# systemctl stop firewalld # 关闭防火墙
另外两台机器相同操作
2.3 其他¶
快速操作
yum install -y ntp \cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime echo "*/5 * * * * /usr/sbin/ntpdate cn.pool.ntp.org > /dev/null 2>&1" >> /var/spool/cron/root cat>>/etc/security/limits.conf<<EOF * soft nofile 65535 * hard nofile 65535 EOF echo "ulimit -SHn 65535">>/etc/rc.local ulimit -SHn 65535 # check getenforce ulimit -SHn crontab -l|grep ntpdate
详细操作
[root@master ~]# yum install -y ntp
imit -SHn 65535">>/etc/rc.local
ulimit -SHn 65535Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* epel: mirror01.idc.hinet.net
* extras: mirrors.163.com
* updates: mirrors.163.com
Package ntp-4.2.6p5-28.el7.centos.x86_64 already installed and latest version
Nothing to do
[root@master ~]# \cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
[root@master ~]# echo "*/5 * * * * /usr/sbin/ntpdate cn.pool.ntp.org > /dev/null 2>&1" >> /var/spool/cron/root
[root@master ~]#
[root@master ~]# cat>>/etc/security/limits.conf<<EOF
> * soft nofile 65535
> * hard nofile 65535
> EOF
[root@master ~]# echo "ulimit -SHn 65535">>/etc/rc.local
[root@master ~]# ulimit -SHn 65535
[root@master ~]# getenforce
Disabled
[root@master ~]# ulimit -SHn
65535
[root@master ~]# timedatectl
Local time: Thu 2019-06-20 17:52:03 CST
Universal time: Thu 2019-06-20 09:52:03 UTC
RTC time: Thu 2019-06-20 09:52:03
Time zone: America/New_York (CST, +0800)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: n/a
[root@master ~]# crontab -l|grep ntpdate
*/5 * * * * /usr/sbin/ntpdate cn.pool.ntp.org > /dev/null 2>&1
其他两个机器都相同操作，省略
2.4 线性关系¶
[root@master ~]# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:fTuySXygaooREU7aggPdTT+DkEtw80es5qJKVCN/Mfo root@master The key's randomart image is: +---[RSA 2048]----+ |..+o++... | |o=.o++.+. | |=.++ +o.= | | o+.+ =. + | | ..o + S o . | |. .+ . o o . | | ... E . + + | |. .o .. . = . | |... .o. o | +----[SHA256]-----+ [root@master ~]# ssh-copy-id master /bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'master (192.168.186.10)' can't be established. ECDSA key fingerprint is SHA256:vcDvqG+xbEjSme1ujEY582F/B3QBpQnxnuenDsLvUb8. ECDSA key fingerprint is MD5:45:a8:62:82:48:1f:6a:7d:fc:3e:3a:60:1a:8d:69:fd. Are you sure you want to continue connecting (yes/no)? yes /bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@master's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'master'" and check to make sure that only the key(s) you wanted were added. [root@master ~]# ssh master Last login: Thu Jun 20 17:30:31 2019 from 192.168.186.1 [root@master ~]# logout [root@master ~]# ssh-copy-id slave1 /bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'slave1 (192.168.186.11)' can't be established. ECDSA key fingerprint is SHA256:E5Y64HSJxuf2Rp9/Ub2eH+UJxH09K/QJyP8PRYXC9qQ. ECDSA key fingerprint is MD5:29:15:f3:14:8b:1c:f4:be:d9:a6:a6:9e:be:27:fa:14. Are you sure you want to continue connecting (yes/no)? yes /bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@slave1's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'slave1'" and check to make sure that only the key(s) you wanted were added. [root@master ~]# ssh slave1 Last login: Thu Jun 20 15:07:09 2019 from 192.168.186.1 [root@slave1 ~]# logout Connection to slave1 closed. [root@master ~]# ssh-copy-id slave2 /bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'slave2 (192.168.186.12)' can't be established. ECDSA key fingerprint is SHA256:ifsSQEkwxLv7+Z1G1J8qe0nrGqJqGUSKwx6iJbosd7Y. ECDSA key fingerprint is MD5:27:7f:ed:8e:e1:98:53:3f:7f:8e:fc:97:4e:19:60:65. Are you sure you want to continue connecting (yes/no)? yes /bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@slave2's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'slave2'" and check to make sure that only the key(s) you wanted were added. [root@master ~]# ssh slave2 Last login: Thu Jun 20 15:07:11 2019 from 192.168.186.1 [root@slave2 ~]# logout Connection to slave2 closed.
每台机器到其他机器都是无密码链接，我只列了master到自己和其他两台机器无密码链接，另外两台机器一样操作。
3. java安装¶
tar xf jdk-8u172-linux-x64.tar.gz -C /usr/local/ cat>>/etc/profile<<EOF JAVA_HOME=/usr/local/jdk1.8.0_172 PATH=\$JAVA_HOME/bin:\$PATH CLASSPATH=.:\$JAVA_HOME/lib/dt.jar:\$JAVA_HOME/lib/tools.jar EOF source /etc/profile java -version
详细过程
[root@localhost local]# tar xf jdk-8u172-linux-x64.tar.gz -C /usr/local/ [root@localhost ~]# cat>>/etc/profile<<EOF > JAVA_HOME=/usr/local/jdk1.8.0_172 > PATH=\$JAVA_HOME/bin:\$PATH > CLASSPATH=.:\$JAVA_HOME/lib/dt.jar:\$JAVA_HOME/lib/tools.jar > EOF [root@localhost local]# tail -3 /etc/profile JAVA_HOME=/usr/local/jdk1.8.0_172 PATH=$JAVA_HOME/bin:$PATH CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar [root@localhost local]# java -version java version "1.8.0_172" Java(TM) SE Runtime Environment (build 1.8.0_172-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)
另外两台机器省略。
4. 安装hadoop¶
tar xf hadoop-2.6.5.tar.gz -C /usr/local/ cd ../hadoop-2.6.5/ mkdir tmp
详细操作
[root@master src]# pwd /usr/local/src [root@master src]# ls hadoop-2.6.5.tar.gz jdk-8u172-linux-x64.tar.gz [root@master src]# tar xf hadoop-2.6.5.tar.gz [root@master src]# tar xf hadoop-2.6.5.tar.gz -C /usr/local
4.1 指定JAVA HOME¶
指定java路径，防止环境变量引起问题 [root@master ~]# cd /usr/local/hadoop-2.6.5/etc/hadoop/ [root@master hadoop]# grep -irn '^export JAVA_HOME=' hadoop-env.sh 26:export JAVA_HOME=/usr/local/jdk1.8.0_172 [root@master hadoop]# grep -irn "JAVA_HOME=/usr/local/jdk1.8.0_172" yarn-env.sh 24:export JAVA_HOME=/usr/local/jdk1.8.0_172 添加从节点主机名 [root@master hadoop]# cat slaves slave1 slave2
4.2 配置core-site.xml¶
添加RPC配置
[root@master hadoop]# cat core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/usr/local/hadoop-2.6.5/tmp</value>
</property>
</configuration>
[root@master hadoop]# mkdir -p /usr/local/hadoop-2.6.5/tmp
hdfs后面添加master的ip或者主机名，file是对应的路径
4.3 配置hdfs-site.xml¶
修改配置文件 添加HDFS配置
[root@master tmp]# mkdir -p /usr/local/hadoop-2.6.5/dfs/name
[root@master tmp]# mkdir -p /usr/local/hadoop-2.6.5/dfs/{name,data}
[root@master tmp]# ls /usr/local/hadoop-2.6.5/dfs/
data name
[root@master hadoop]# cat hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop-2.6.5/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop-2.6.5/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
4.4 配置mapred-site.xml¶
[root@master hadoop]# cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
4.5 配置yarn-site.xml¶
添加资源管理配置
[root@master hadoop]# cat yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8035</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
4.6 添加hadoop环境变量¶
[root@master hadoop]# cat>>/etc/profile<<EOF > # add by caimengzhi > HADOOP_HOME=/usr/local/hadoop-2.6.5 > export PATH=\$PATH:\$HADOOP_HOME/bin > EOF [root@master hadoop]# tail -3 /etc/profile # add by caimengzhi HADOOP_HOME=/usr/local/hadoop-2.6.5 export PATH=$PATH:$HADOOP_HOME/bin [root@master hadoop]# source /etc/profile
4.7 拷贝¶
- 拷贝环境变量配置文件到从
[root@master hadoop]# scp /etc/profile root@slave1:/etc/profile [root@master hadoop]# scp /etc/profile root@slave2:/etc/profile
- 拷贝hadoop源码包到从
[root@master hadoop]# rsync -az /usr/local/hadoop-2.6.5 root@slave1:/usr/local/ [root@master hadoop]# rsync -az /usr/local/hadoop-2.6.5 root@slave2:/usr/local/
- 重新加载环境变量
[root@slave1 ~]# source /etc/profile [root@slave2 ~]# source /etc/profile
4.8 格式化¶
[root@master hadoop]# hadoop namenode -format DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. 19/06/20 19:41:00 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = master/192.168.186.10 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.6.5 STARTUP_MSG: classpath = /usr/local/hadoop-2.6.5/etc/hadoop:/usr/local/hadoop-2.6.5/share/hadoop/common/li b/activation-1.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/htrace-core-3.0.4.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/hadoop-annotations-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/curator-framework-2.6.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-el-1.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/hadoop-auth-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/curator-client-2.6.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/hadoop-common-2.6.5-tests.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/hadoop-nfs-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/common/hadoop-common-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/hadoop-hdfs-2.6.5-tests.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/hadoop-hdfs-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-common-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-api-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-registry-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-client-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-common-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.5-tests.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.5.jar:/usr/local/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.5.jar:/contrib/capacity-scheduler/*.jar:/contrib/capacity-scheduler/*.jarSTARTUP_MSG: build = https://github.com/apache/hadoop.git -r e8c9fe0b4c252caf2ebf1464220599650f119997; com piled by 'sjlee' on 2016-10-02T23:43ZSTARTUP_MSG: java = 1.8.0_172 ************************************************************/ 19/06/20 19:41:00 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 19/06/20 19:41:00 INFO namenode.NameNode: createNameNode [-format] Formatting using clusterid: CID-75d915c8-2079-4742-89d8-551fb15be6ae 19/06/20 19:41:00 INFO namenode.FSNamesystem: No KeyProvider found. 19/06/20 19:41:00 INFO namenode.FSNamesystem: fsLock is fair:true 19/06/20 19:41:00 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 19/06/20 19:41:00 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check =true19/06/20 19:41:00 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.00019/06/20 19:41:00 INFO blockmanagement.BlockManager: The block deletion will start around 2019 Jun 20 19:41: 0019/06/20 19:41:00 INFO util.GSet: Computing capacity for map BlocksMap 19/06/20 19:41:00 INFO util.GSet: VM type = 64-bit 19/06/20 19:41:00 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB 19/06/20 19:41:00 INFO util.GSet: capacity = 2^21 = 2097152 entries 19/06/20 19:41:00 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 19/06/20 19:41:00 INFO blockmanagement.BlockManager: defaultReplication = 3 19/06/20 19:41:00 INFO blockmanagement.BlockManager: maxReplication = 512 19/06/20 19:41:00 INFO blockmanagement.BlockManager: minReplication = 1 19/06/20 19:41:00 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 19/06/20 19:41:00 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 19/06/20 19:41:00 INFO blockmanagement.BlockManager: encryptDataTransfer = false 19/06/20 19:41:00 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 19/06/20 19:41:00 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE) 19/06/20 19:41:00 INFO namenode.FSNamesystem: supergroup = supergroup 19/06/20 19:41:00 INFO namenode.FSNamesystem: isPermissionEnabled = true 19/06/20 19:41:00 INFO namenode.FSNamesystem: HA Enabled: false 19/06/20 19:41:00 INFO namenode.FSNamesystem: Append Enabled: true 19/06/20 19:41:01 INFO util.GSet: Computing capacity for map INodeMap 19/06/20 19:41:01 INFO util.GSet: VM type = 64-bit 19/06/20 19:41:01 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB 19/06/20 19:41:01 INFO util.GSet: capacity = 2^20 = 1048576 entries 19/06/20 19:41:01 INFO namenode.NameNode: Caching file names occuring more than 10 times 19/06/20 19:41:01 INFO util.GSet: Computing capacity for map cachedBlocks 19/06/20 19:41:01 INFO util.GSet: VM type = 64-bit 19/06/20 19:41:01 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB 19/06/20 19:41:01 INFO util.GSet: capacity = 2^18 = 262144 entries 19/06/20 19:41:01 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 19/06/20 19:41:01 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 19/06/20 19:41:01 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 19/06/20 19:41:01 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 19/06/20 19:41:01 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis19/06/20 19:41:01 INFO util.GSet: Computing capacity for map NameNodeRetryCache 19/06/20 19:41:01 INFO util.GSet: VM type = 64-bit 19/06/20 19:41:01 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB 19/06/20 19:41:01 INFO util.GSet: capacity = 2^15 = 32768 entries 19/06/20 19:41:01 INFO namenode.NNConf: ACLs enabled? false 19/06/20 19:41:01 INFO namenode.NNConf: XAttrs enabled? true 19/06/20 19:41:01 INFO namenode.NNConf: Maximum size of an xattr: 16384 19/06/20 19:41:01 INFO namenode.FSImage: Allocated new BlockPoolId: BP-620267902-192.168.186.10-156103086107 019/06/20 19:41:01 INFO common.Storage: Storage directory /usr/local/hadoop-2.6.5/dfs/name has been successfu lly formatted.19/06/20 19:41:01 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/hadoop-2.6.5/dfs/name/cu rrent/fsimage.ckpt_0000000000000000000 using no compression19/06/20 19:41:01 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/hadoop-2.6.5/dfs/name/current/f simage.ckpt_0000000000000000000 of size 321 bytes saved in 0 seconds.19/06/20 19:41:01 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 19/06/20 19:41:01 INFO util.ExitUtil: Exiting with status 0 19/06/20 19:41:01 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at master/192.168.186.10 ************************************************************/
提示下列内容即是成功完成格式化
common.Storage: Storage directory /usr/local/hadoop-2.6.5/dfs/name has been successfu
5. 启动集群¶
5.1 启动集群¶
[root@master bin]# /usr/local/hadoop-2.6.5/sbin/start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [master] master: starting namenode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-namenode-master.out slave2: starting datanode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-slave2.out slave1: starting datanode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-slave1.out Starting secondary namenodes [master] master: starting secondarynamenode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-secondarynamenode-master.out starting yarn daemons starting resourcemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-resourcemanager-master.out slave2: starting nodemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-nodemanager-slave2.out slave1: starting nodemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-nodemanager-slave1.out
在master上启动集群，master会通过ssh去启动slave进程
启动: /usr/local/hadoop-2.6.5/sbin/start-all.sh
停止: /usr/local/hadoop-2.6.5/sbin/stop-all.sh
5.2 查看服务进程¶
- master
[root@master bin]# jps 7987 SecondaryNameNode 7812 NameNode 8136 ResourceManager 8395 Jps
- slave
[root@slave1 ~]# jps 7424 Jps 7215 DataNode 7311 NodeManager [root@slave2 ~]# jps 7416 DataNode 7512 NodeManager 7624 Jps
5.3 测试hdfs¶
上传文件到hdfs文件系统中 [root@master bin]# hadoop fs -put /etc/passwd / 查看hdfs文件系统中的文件 [root@master bin]# hadoop fs -ls / Found 1 items -rw-r--r-- 3 root supergroup 944 2019-06-20 19:50 /passwd 读取文件内容 [root@master bin]# hadoop fs -text /passwd | tail -3 chrony:x:998:996::/var/lib/chrony:/sbin/nologin ntp:x:38:38::/etc/ntp:/sbin/nologin saslauth:x:997:76:Saslauthd user:/run/saslauthd:/sbin/nologin
5.4 web端测试¶
hosts文件中添加三个主机的名和ip
192.168.186.10 master 192.168.186.11 slave1 192.168.186.12 slave2
windows
C:\Windows\System32\drivers\etc\hostsLinux
/etc/hosts要在测试的客户端机器上添加这个hosts解析
浏览器输入http://master:50070 就可以看到后台hdfs数据信息
监控页面 http://master:8088
| Daemon | Web Interface | Notes |
|---|---|---|
| NameNode | http://nn_host:port/ | Default HTTP port is 50070. |
| ResourceManager | http://rm_host:port/ | Default HTTP port is 8088. |
| MapReduce JobHistory Server | http://jhs_host:port/ | Default HTTP port is 19888. |