當(dāng)前位置：首頁(yè) > 编程资源 > 编程问答 >内容正文

编程问答

hadoop 多机全分布式安装步骤（虚拟机1master+2slave）

發(fā)布時(shí)間：2024/7/5 编程问答 24 豆豆

生活随笔收集整理的這篇文章主要介紹了 hadoop 多机全分布式安装步骤（虚拟机1master+2slave）小編覺(jué)得挺不錯(cuò)的,現(xiàn)在分享給大家,幫大家做個(gè)參考.

文章目錄

- 1. 虛擬機(jī)安裝Centos7
- 2. 配置靜態(tài)IP
- 3. 更改主機(jī)名
- 4. 編輯域名映射
- 5. 安裝配置Java
- 6. 配置SSH免密登錄
- 7 .安裝Hadoop
- 8. 關(guān)閉防火墻
- 9. 格式化文件系統(tǒng)
- 10. 啟動(dòng)驗(yàn)證
- 11. 第一個(gè)MapReduce程序: WordCount
- 12. 關(guān)閉Hadoop

參考書：《Hadoop大數(shù)據(jù)原理與應(yīng)用》

1. 虛擬機(jī)安裝Centos7

安裝3臺(tái)虛擬機(jī)，centos7，一個(gè)master，兩個(gè)slave，安裝時(shí)可以改hostname, 記得設(shè)置密碼
安裝的是4.7Gb的包，選擇的 service with GUI
選則 NAT 網(wǎng)絡(luò)鏈接
ip route show 查看路由器網(wǎng)關(guān)ip
ip addr 查找本機(jī)ip（下面用的著這兩個(gè)ip）

2. 配置靜態(tài)IP

vim /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=static # 改靜態(tài) DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens33 UUID=caf90547-4b5a-46b3-ab7c-2c8fb1f5e4d7 DEVICE=ens33 ONBOOT=yes # 改yesIPADDR=192.168.253.130 # ip NETMASK=255.255.255.0 GATEWAY=192.168.253.2 # 網(wǎng)關(guān) DNS1=192.168.253.2 # 跟網(wǎng)關(guān)一樣即可

保存權(quán)限不足，輸入w !sudo tee %

重啟網(wǎng)絡(luò)

systemctl restart network.service

2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000link/ether 00:0c:29:77:14:71 brd ff:ff:ff:ff:ff:ffinet 192.168.253.130/24 brd 192.168.253.255 scope global noprefixroute ens33valid_lft forever preferred_lft foreverinet6 fe80::ce06:b26c:ff2d:5288/64 scope link noprefixroute valid_lft forever preferred_lft forever

同理，另外兩臺(tái) ip 為：192.168.253.128, 192.168.253.129（個(gè)人根據(jù)自己的情況來(lái)）

3. 更改主機(jī)名

安裝的時(shí)候就改了，此處可跳過(guò)
切換 root 用戶，sudo su
vi /etc/hostname, 分別替換內(nèi)容為 master，slave1， slave2
reboot重啟，hostname 查看是否更改

4. 編輯域名映射

為了便捷訪問(wèn)，三臺(tái)機(jī)器都做以下修改，sudo su
在 /etc/hosts追加以下內(nèi)容，重啟

192.168.253.130 master 192.168.253.128 slave1 192.168.253.129 slave2 192.168.31.237 michael

檢查各臺(tái)機(jī)器是否能ping通

ping master ping slave1 ping slave2

5. 安裝配置Java

卸載
查看 java -version
卸載自帶的 Oracle OpenJDK，使用Oracle JDK

rpm -qa | grep jdk yum -y remove java-1.8.0* yum -y remove java-1.7.0*

下載 jdk，位數(shù)根據(jù)下圖來(lái)
我從宿主機(jī)直接考過(guò)來(lái)安裝包

scp root@michael:/home/dnn/jdk-8u281-linux-x64.tar.gz /home/dnn/jdk-8u281-linux-x64.tar.gz

參考 JDK 安裝
裝到/opt/jdk1.8.0_281/

6. 配置SSH免密登錄

查詢 rpm -qa | grep ssh
沒(méi)有的話安裝

yum -y install openssh yum -y install openssh-server yum -y install openssh-clients

vim /etc/ssh/sshd_config
第43行取消注釋，并加一行，3臺(tái)機(jī)器都做

RSAAuthentication yes PubkeyAuthentication yes

systemctl restart sshd.service，重啟服務(wù)
切換普通用戶 ctrl+d，回到home cd ~
ssh-keygen,一直回車
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
將master的公鑰復(fù)制給slave1，slave2，免密訪問(wèn)從節(jié)點(diǎn)

在master里鍵入以下命令

ssh-copy-id -i ~/.ssh/id_rsa.pub dnn@slave1 ssh slave1 ssh-copy-id -i ~/.ssh/id_rsa.pub dnn@slave2 ssh slave2 ssh master

遇到提示輸入 yes，敲密碼

還可以在另外兩臺(tái)里，同樣的步驟操作一遍

7 .安裝Hadoop

偽分布式可以參考：hadoop 單機(jī)偽分布式安裝步驟

下載或拷貝安裝包到3臺(tái)機(jī)器

scp dnn@michael:/home/dnn/hadoop-3.3.0.tar.gz /home/dnn/hadoop-3.3.0.tar.gz

到文件目錄下，解壓 tar -zxvf hadoop-3.3.0.tar.gz
移動(dòng)到你要放的目錄 sudo mv hadoop-3.3.0 /opt/hadoop-3.3.0
賦權(quán)限給普通用戶dnn，chown -R dnn /opt/hadoop-3.3.0

在主節(jié)點(diǎn)上操作：

切換 root 用戶，新建文件 vim /etc/profile.d/hadoop.sh
添加內(nèi)容

export HADOOP_HOME=/opt/hadoop-3.3.0 export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH source /etc/profile.d/hadoop.sh

切換普通用戶，上面已賦權(quán)限，vim /opt/hadoop-3.3.0/etc/hadoop/hadoop-env.sh
54行改為 export JAVA_HOME=/opt/jdk1.8.0_281/
55行添加 export HADOOP_SSH_OPTS='-o StrictHostKeyChecking=no'
199行修改 export HADOOP_PID_DIR=${HADOOP_HOME}/pids
vim /opt/hadoop-3.3.0/etc/hadoop/mapred-env.sh
添加 export JAVA_HOME=/opt/jdk1.8.0_281/，export HADOOP_MAPRED_PID_DIR=${HADOOP_HOME}/pids
vim /opt/hadoop-3.3.0/etc/hadoop/yarn-env.sh
添加

export JAVA_HOME=/opt/jdk1.8.0_281/ export YARN_PID_DIR=${HADOOP_HOME}/pids

vim /opt/hadoop-3.3.0/etc/hadoop/core-site.xml

<configuration><property><name>fs.defaultFS</name><value>hdfs://192.168.253.130:9000</value></property><property><name>hadoop.tmp.dir</name><value>/opt/hadoop-3.3.0/hdfsdata</value></property><property><name>io.file.buffer.size</name><value>131072</value></property> <configuration>

vim /opt/hadoop-3.3.0/etc/hadoop/mapred-site.xml

<configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property> </configuration>

vim /opt/hadoop-3.3.0/etc/hadoop/yarn-site.xml

<configuration><property><name>yarn.resourcemanager.hostname</name><value>master</value></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property> </configuration>

在 /opt/hadoop-3.3.0/etc/hadoop/ 下，vim workers
刪除 localhost，加入

slave1 slave2

同步配置文件到2臺(tái)slave上

在root下

scp /etc/profile.d/hadoop.sh root@slave1:/etc/profile.d/ scp /etc/profile.d/hadoop.sh root@slave2:/etc/profile.d/

在普通用戶下

scp -r /opt/hadoop-3.3.0/etc/hadoop/* dnn@slave1:/opt/hadoop-3.3.0/etc/hadoop/ scp -r /opt/hadoop-3.3.0/etc/hadoop/* dnn@slave2:/opt/hadoop-3.3.0/etc/hadoop/

8. 關(guān)閉防火墻

在root下，systemctl disable firewalld.service
重啟，再看下狀態(tài) systemctl status firewalld.service
顯示 inactive(dead), 3臺(tái)機(jī)器都做

9. 格式化文件系統(tǒng)

只在 master 上用普通用戶操作：

hdfs namenode -format

10. 啟動(dòng)驗(yàn)證

在 master 上執(zhí)行3條命令

start-dfs.sh start-yarn.sh mr-jobhistory-daemon.sh start historyserver # 第三條可以用下面的命令，上面的顯示過(guò)期了，以后棄用 mapred --daemon start historyserver

輸入 jps 命令，可以看見(jiàn)進(jìn)程啟動(dòng)了

11. 第一個(gè)MapReduce程序: WordCount

master：在HDFS根目錄下創(chuàng)建目錄

[dnn@master ~]$ hdfs dfs -mkdir /InputDataTest [dnn@master ~]$ hdfs dfs -ls / Found 2 items drwxr-xr-x - dnn supergroup 0 2021-03-12 06:49 /InputDataTest drwxrwx--- - dnn supergroup 0 2021-03-12 06:19 /tmp

上傳文件到 InputDataTest 文件夾

[dnn@master ~]$ hdfs dfs -put /opt/hadoop-3.3.0/etc/hadoop/hadoop-env.sh /InputDataTest [dnn@master ~]$ hdfs dfs -put /opt/hadoop-3.3.0/etc/hadoop/mapred-env.sh /InputDataTest [dnn@master ~]$ hdfs dfs -put /opt/hadoop-3.3.0/etc/hadoop/yarn-env.sh /InputDataTest [dnn@master ~]$ hdfs dfs -ls /InputDataTest Found 3 items -rw-r--r-- 3 dnn supergroup 17017 2021-03-12 06:52 /InputDataTest/hadoop-env.sh -rw-r--r-- 3 dnn supergroup 1850 2021-03-12 06:53 /InputDataTest/mapred-env.sh -rw-r--r-- 3 dnn supergroup 6406 2021-03-12 06:53 /InputDataTest/yarn-env.sh

hadoop jar /opt/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar wordcount /InputDataTest /OutputDataTest
報(bào)錯(cuò)： org.apache.hadoop.mapreduce.v2.app.MRAppMaster

vim /opt/hadoop-3.3.0/etc/hadoop/yarn-site.xml <configuration><property><name>yarn.resourcemanager.hostname</name><value>master</value></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.application.classpath</name><value>/opt/hadoop-3.3.0/etc/hadoop:/opt/hadoop-3.3.0/share/hadoop/common/lib/*:/opt/hadoop-3.3.0/share/hadoop/common/*:/opt/hadoop-3.3.0/share/hadoop/hdfs:/opt/hadoop-3.3.0/share/hadoop/hdfs/lib/*:/opt/hadoop-3.3.0/share/hadoop/hdfs/*:/opt/hadoop-3.3.0/share/hadoop/mapreduce/*:/opt/hadoop-3.3.0/share/hadoop/yarn:/opt/hadoop-3.3.0/share/hadoop/yarn/lib/*:/opt/hadoop-3.3.0/share/hadoop/yarn/* </value></property> </configuration>

重啟集群（關(guān)閉3條命令，見(jiàn)下面第12節(jié)，啟動(dòng)3條命令），再次運(yùn)行 wordcount 程序

[dnn@master ~]$ hadoop jar /opt/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar wordcount /InputDataTest /OutputDataTest 2021-03-12 07:11:51,635 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at master/192.168.253.130:8032 2021-03-12 07:11:52,408 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/dnn/.staging/job_1615504213995_0001 2021-03-12 07:11:53,547 INFO input.FileInputFormat: Total input files to process : 3 2021-03-12 07:11:54,066 INFO mapreduce.JobSubmitter: number of splits:3 2021-03-12 07:11:54,271 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1615504213995_0001 2021-03-12 07:11:54,271 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2021-03-12 07:11:54,624 INFO conf.Configuration: resource-types.xml not found 2021-03-12 07:11:54,624 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2021-03-12 07:11:55,117 INFO impl.YarnClientImpl: Submitted application application_1615504213995_0001 2021-03-12 07:11:55,164 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1615504213995_0001/ 2021-03-12 07:11:55,164 INFO mapreduce.Job: Running job: job_1615504213995_0001 2021-03-12 07:12:05,308 INFO mapreduce.Job: Job job_1615504213995_0001 running in uber mode : false 2021-03-12 07:12:05,319 INFO mapreduce.Job: map 0% reduce 0% 2021-03-12 07:12:21,455 INFO mapreduce.Job: map 33% reduce 0% 2021-03-12 07:12:22,460 INFO mapreduce.Job: map 100% reduce 0% 2021-03-12 07:12:29,514 INFO mapreduce.Job: map 100% reduce 100% 2021-03-12 07:12:29,526 INFO mapreduce.Job: Job job_1615504213995_0001 completed successfully 2021-03-12 07:12:29,652 INFO mapreduce.Job: Counters: 54File System CountersFILE: Number of bytes read=20470FILE: Number of bytes written=1097885FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=25631HDFS: Number of bytes written=12134HDFS: Number of read operations=14HDFS: Number of large read operations=0HDFS: Number of write operations=2HDFS: Number of bytes read erasure-coded=0Job Counters Launched map tasks=3Launched reduce tasks=1Data-local map tasks=3Total time spent by all maps in occupied slots (ms)=42362Total time spent by all reduces in occupied slots (ms)=4808Total time spent by all map tasks (ms)=42362Total time spent by all reduce tasks (ms)=4808Total vcore-milliseconds taken by all map tasks=42362Total vcore-milliseconds taken by all reduce tasks=4808Total megabyte-milliseconds taken by all map tasks=43378688Total megabyte-milliseconds taken by all reduce tasks=4923392Map-Reduce FrameworkMap input records=667Map output records=3682Map output bytes=39850Map output materialized bytes=20482Input split bytes=358Combine input records=3682Combine output records=1261Reduce input groups=912Reduce shuffle bytes=20482Reduce input records=1261Reduce output records=912Spilled Records=2522Shuffled Maps =3Failed Shuffles=0Merged Map outputs=3GC time elapsed (ms)=800CPU time spent (ms)=2970Physical memory (bytes) snapshot=615825408Virtual memory (bytes) snapshot=10951270400Total committed heap usage (bytes)=385785856Peak Map Physical memory (bytes)=168960000Peak Map Virtual memory (bytes)=2738552832Peak Reduce Physical memory (bytes)=110534656Peak Reduce Virtual memory (bytes)=2742329344Shuffle ErrorsBAD_ID=0CONNECTION=0IO_ERROR=0WRONG_LENGTH=0WRONG_MAP=0WRONG_REDUCE=0File Input Format Counters Bytes Read=25273File Output Format Counters Bytes Written=12134

查看結(jié)果

[dnn@master ~]$ hdfs dfs -ls /OutputDataTest Found 2 items -rw-r--r-- 3 dnn supergroup 0 2021-03-12 07:12 /OutputDataTest/_SUCCESS -rw-r--r-- 3 dnn supergroup 12134 2021-03-12 07:12 /OutputDataTest/part-r-00000

_SUCCESS 表示運(yùn)行成功
結(jié)果文件是 part-r-00000

hdfs dfs -cat /OutputDataTest/part-r-00000 查看結(jié)果

[dnn@master ~]$ hdfs dfs -cat /OutputDataTest/part-r-00000 "AS 3 "License"); 3 "log 1 # 466 ## 32 ### 53 #export 14 $HADOOP_YARN_HOME/share/hadoop/yarn/yarn-service-examples 1 $USER 1 ${HADOOP_HOME}/logs 1 ${HOME}/.hadooprc 1 '-' 1 '.' 1 'hadoop 1 'mapred 1 'yarn 1 ( 1 (ASF) 3 (BUT 1 (Java 2 (Note 1 (command)_(subcommand)_USER. 1 (e.g., 1 (file/dir 1 (i.e., 2 (period) 1 (primarily) 1 (such 1 (superficially) 1 (the 3 ) 1 **MUST 1 **MUST** 1 *NOT* 1 +'%Y%m%d%H%M')" 4 --config) 1 --daemon 1 -Dcom.sun.management.jmxremote.authenticate=false 2 -Dcom.sun.management.jmxremote.port=1026" 2 -Dcom.sun.management.jmxremote.ssl=false 2 -Dhadoop.security.logger=foo). 1 -Dsun.security.krb5.debug=true 1 -Dsun.security.spnego.debug" 1 -Dyarn.server.resourcemanager.appsummary.logger=INFO,RMSUMMARY" 1 -XX:+PrintGCDateStamps 2 -XX:+PrintGCDateStamps" 1 -XX:+PrintGCDetails 3 -XX:+PrintGCTimeStamps 3 -Xloggc:${HADOOP_LOG_DIR}/gc-rm.log-$(date 4 -Xms). 1 -Xmx). 1 -blah). 1 -f 1 -ls 1 -o 2 -s)} 1 . 1 .. 1 ... 2 .hadooprc 1 /etc/profile.d 2 /tmp 1 10 1 2.0 3 2NN 1 <----- 1 = 1 > 12

12. 關(guān)閉Hadoop

mr-jobhistory-daemon.sh stop historyserver # 或者 mapred --daemon stop historyserver stop-yarn.sh stop-dfs.sh

好幾天了，跟著書，終于安裝成功了！

我的CSDN博客地址 https://michael.blog.csdn.net/

長(zhǎng)按或掃碼關(guān)注我的公眾號(hào)（Michael阿明），一起加油、一起學(xué)習(xí)進(jìn)步！

總結(jié)

以上是生活随笔為你收集整理的hadoop 多机全分布式安装步骤（虚拟机1master+2slave）的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。

上一篇： LeetCode 1976. 到达目的地
下一篇： LeetCode 2049. 统计最高分