當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

CDH Hadoop 基于CM方式半在线安装步骤详解

發布時間：2024/9/27 编程问答 19 豆豆

生活随笔收集整理的這篇文章主要介紹了 CDH Hadoop 基于CM方式半在线安装步骤详解小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

CDH Hadoop基于Cloudera Manager半在線方式安裝詳解

1 CM簡介

Cloudera Manager是一個端到端的管理CDH集群的工具。它通過管理CDH集群（組件安裝、服務管理、參數監控、問題診斷、性能監控）各個環節從而增強集群管理的質量, 增強規范性,降低運營成本。

注:CM自6.3.3之后不再支持社區免費版(Express),后續版本需付費才能下載和使用。

2 準備

2.1 環境說明

1 該安裝方式依賴如下linux組件:httpd, mysql

2 安裝時需要用到其它系統組件,請配置好本地源

3 安裝mysql是請確保清理好系統自帶的mysql.

4 CM 下載地址見下,需要下載如下的所有文件.詳細目錄分布見2.2.6說明.

http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1/RPMS/x86_64/

http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1/repodata/

http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera

http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo

5 CDH parcel下載地址http://archive-primary.cloudera.com/cdh5/parcels/5.5.1/

6 CM installer下載地址

http://archive-primary.cloudera.com/cm5/installer/5.5.1/cloudera-manager-installer.bin

注:

1 CM 核心安裝文件截圖:

2 CDH parcel相關文件截圖:

3 CM installer截圖:

2.1.1 虛擬機里Linux系統版本

[root@localhost ranger-0.5.0-usersync]# cat /etc/issue | grep Linux

Red Hat Enterprise Linux Server release 6.5 (Santiago)

2.1.2 JDK版本

[root@localhost native]# java -version

java version "1.7.0_67"

注:官網強調一定是1.7以上版本.

Java(TM) SE RuntimeEnvironment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-Bit ServerVM (build 24.65-b04, mixed mode)

2.1.3mysql版本

[root@localhost native]# mysql -uroot -proot-e"select version()";

Warning: Using a password onthe command line interface can be insecure.

+-----------+

| version() |

+-----------+

| 5.6.14 ???|

+-----------+

注:

1 Mysql 驅動為mysql-connector-java-5.1.31-bin.jar

2 該jar需要放置在/usr/share/cmf/lib/內被CM安裝時用到

2.2 安裝準備

2.2.1 禁用防火墻

service iptables stop

chkconfig iptables off

2.2.2 禁用Selinux

vi /etc/selinux/config

#修改內容見下:

SELINUX=disabled

SELINUXTYPE=targeted

2.2.3 修改主機名

vi/etc/hosts

#加上archive.cloudera.com域名

192.168.56.101 archive.cloudera.com shenl

vi/etc/sysconfig/network

#修改主機名如下:

HOSTNAME=archive.cloudera.com

#重啟電腦使得更改生效.

2.2.4 配置本地源

1) 虛擬機加載Linux ISO鏡像

2) 找到rom對應設備名

[root@localhost ~]# lsblk

NAME??????????????????????? MAJ:MIN RM? SIZE RO TYPE MOUNTPOINT

sr0????????????????????????? 11:0??? 1 1024M?0 rom?

sr1????????????????????????? 11:1??? 1?3.6G? 0 rom?

sr2????????????????????????? 11:2??? 1 1024M?0 rom?

sda?????????????????????????? 8:0??? 0???8G? 0 disk

?..sda1??????????????????????? 8:1??? 0?500M? 0 part /boot

?..sda2??????????????????????? 8:2??? 0?7.5G? 0 part

??..VolGroup-lv_root (dm-0) 253:0???0? 6.7G? 0 lvm?/

??..VolGroup-lv_swap (dm-1) 253:1???0? 816M? 0 lvm?[SWAP]

3) 創建目錄

[root@localhost ~]#mkdir -p /mnt/cdrom/

4) 掛載鏡像

mount -t iso9660 /dev/sr1 /mnt/cdrom

5) 配置linux本地更新源(真實環境這里的應該baseurl應配置成http方式,如http://ios)

vi /etc/yum.repos.d/redhat.repo

#修改內容如下:

[rhel-source]

name=Redhat

baseurl=file:///mnt/cdrom/

enabled=1

gpgcheck=1

gpgkey=file:///mnt/cdrom//RPM-GPG-KEY-redhat-release

6) 更新更新源

yum clean all

yum makecache

yum update list

2.2.5 安裝配置mysql

1)? 刪除系統自帶的mysql

rpm -qa | grep -i mysql

MySQL-client-5.1.49-1.glibc23

rpm -e --nodeps MySQL-client-5.1.49-1.glibc23

chkconfig --list | grep -i mysql?

mysql??????????0:off?? 1:off?? 2:on???3:on??? 4:on??? 5:on???6:off?

chkconfig --del mysql

whereis mysql?

mysql: /usr/lib/mysql /usr/share/mysql ?

rm -rf /usr/lib/mysql/?

rm -rf /usr/share/mysql

2) 安裝Mysql服務、客戶端

rpm -ivh MySQL-shared-5.6.14-1.el6.x86_64.rpm

rpm -ivh MySQL-shared-compat-5.6.14-1.el6.x86_64.rpm

rpm -ivh MySQL-server-5.6.14-1.el6.x86_64.rpm

rpm -ivh MySQL-client-5.6.14-1.el6.x86_64.rpm

3) 啟動mysql服務

service mysql start

4) 修改mysql初始密碼,先找到安裝時的初始密碼,在修改成自己的密碼

[root@localhost ~]#cat /root/.mysql_secret

# The random password set for the root user at Tue Dec 2221:17:22 2015 (local time):RUmKBqcY

mysql -uroot -p RUmKBqcY

set password=password(‘root’)

注:??如果想修改mysql默認的datadir,假設到/data/mysqldata內,可參考如下方式修改:

#遷移mysql數據目錄,做軟連接,修改目錄的所有者

mv /var/lib/mysql//data/mysqldata/mysql????????????????

ln -s /data/mysqldata/mysql/var/lib/

chown -R mysql:mysql/data/mysqldata/mysql

2.2.6 安裝httpd,并按照如下的目錄結構放置CM和CDH parcel的安裝文件.

yum install tree

yum install httpd

service httpd start

cd /var/www/html

注:如果想修改httpd的網站默認主目錄,可通過如下方式:

mkdir -p /data/www/

chown -R apache:apache/data/www/

vi /etc/httpd/conf/httpd.conf

#找到DocumentRoot 修改"/var/www/html"成"/data/www"

service httpd restart

報如下錯:

Forbidden

You don't have permission to access /cm/ on this server.

vi /etc/httpd/conf/httpd.conf

#找到 <Directory"/var/www/html"> 修改成 /data/www

service httpd restart

#CM的目錄安排需要按照如下方式:

[root@archive html]# tree./cm5/

./cm5/

└── redhat

???└── 6

???????└── x86_64

??????????? └── cm

??????????????? ├── 5.5.1

??????????????? │??├── repodata

??????????????? │??│??├── filelists.xml.gz

??????????????? │??│??├── filelists.xml.gz.asc

??????????????? │??│??├── other.xml.gz

??????????????? │??│??├── other.xml.gz.asc

??????????????? │??│??├── primary.xml.gz

??????????????? │??│??├── primary.xml.gz.asc

??????????????? │??│??├── repomd.xml

??????????????? │??│??└── repomd.xml.asc

??????????????? │??└── RPMS

??????????????? │??????└── x86_64

??????????????? │??????????├──cloudera-manager-agent-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

???????????????│??????????├── cloudera-manager-daemons-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

??????????????? │??????????├──cloudera-manager-server-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

??????????????? │??????????├──cloudera-manager-server-db-2-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

??????????????? │??????????├──enterprise-debuginfo-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

??????????????? │??????????├── jdk-6u31-linux-amd64.rpm

??????????????? │??????????└──oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

??????????????? ├── cloudera-manager.repo

??????????????? └── RPM-GPG-KEY-cloudera

8 directories, 17 files

#CDH parcel的目錄安排需要按照如下方式:

[root@archive html]#tree ./cdh5/

./cdh5/

└── parcels

???└── 5.5.1

???????├── CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel

???????├── CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel.sha1

???????└── manifest.json

2 directories, 3 files

#將CM安裝腳本放置/var/www/html(可放置在其它目錄,這里為了統一).:

chmod? +x cloudera-manager-installer.bin

注:下載repodata文件夾時可以在Window里下載好,執行wget命令

E:\ProgramFiles\wget-1.16.3-win64>wget -r -np -ndhttp://archive-primary.cloude

ra.com/cm5/redhat/6/x86_64/cm/5.5.1/repodata/

3安裝

3.1安裝中

1) 執行CM安裝腳本

./cloudera-manager-installer.bin

#分析并解決CM安裝過程中的問題,詳細見 3.2安裝問題:

cat/var/log/cloudera-manager-installer/2.install-oracle-j2sdk1.7.log

2) 輸入CM所在的IP地址和7180端口,進行界面化安裝CDH.

?#選擇存儲庫時需要點擊parcel旁邊的更多選項,刪除默認定義的路徑,寫上

http://192.168.56.101/cdh5/parcels/5.5.1

或者

http://archive.cloudera.com/cdh5/parcels/5.5.1

#這時如果選擇此Cloudera Manager Server的匹配發行版,會報錯,想見3.2安裝問題之d).這里建議選擇自定義存儲庫.見下圖:

2)? 按照提示一步步安裝,直至集群安裝成功…

注: CM在安裝時會自動生成cloudera-manager.repo,請確認里面的url跟自定義存儲庫里的一致,建議走http協議.cloudera-manager.repo里默認下載地址見下:

https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1

3.2 安裝問題

1) 正確配置CM和CDH的目錄樹結構

2) 正確配置parcel和自定義存儲庫

a) Loaded plugins: product-id,security, subscription-manager

This system is not registered to Red Hat SubscriptionManagement. You can use subscription-manager to register.

http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/repodata/repomd.xml:[Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 NotFound"

解決方案:重命名文件名,詳細見下:

cd /var/www/html

mv ./cm5/redhat/6/x86_64/cm/5.5.1./cm5/redhat/6/x86_64/cm/5

b) .

解決方案:重命名文件名,詳細見如下:

mv ./cm5/redhat/6/x86_64/cm/5 ./cm5/redhat/6/x86_64/cm/5.5.1

c) CMinstaller正常安裝相關日志

?[root@archive cloudera-manager-installer]# pwd

/var/log/cloudera-manager-installer

[root@archivecloudera-manager-installer]# ll

total 20

-rw-r--r-- 1 root root??? 0 Jan?4 15:51 0.check-selinux.log

-rw-r--r-- 1 root root??? 0 Jan?4 15:51 1.install-repo-pkg.log

-rw-r--r-- 1 root root 1647 Jan? 4 15:51 2.install-oracle-j2sdk1.7.log

-rw-r--r-- 1 root root 2080 Jan? 4 15:53 3.install-cloudera-manager-server.log

-rw-r--r-- 1 root root??? 0 Jan?4 15:46 3.remove-cloudera-manager-repository.log

-rw-r--r-- 1 root root 3250 Jan? 4 15:53 4.install-cloudera-manager-server-db-2.log

-rw-r--r-- 1 root root 1978 Jan? 4 15:54 5.start-embedded-db.log

-rw-r--r-- 1 root root?? 59 Jan?4 15:54 6.start-scm-server.log

d) 按照錯誤提示修改成b)后仍然報如下錯誤

BEGIN yum info cloudera-manager-agent?
Loaded plugins: product-id, security, subscription-manager?
This system is not registered to Red Hat Subscription Management. You can usesubscription-manager to register.?
https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1/repodata/repomd.xml:[Errno 14] PYCURL ERROR 7 - "couldn't connect to host"?
Trying other mirror.?
Error: Cannot retrieve repository metadata (repomd.xml) for repository:cloudera-manager. Please verify its path and try again

解決方案:這時需要在群集安裝→選擇存儲庫里選擇自定義存儲庫,配置見下圖:

e) 配置CM事件監控、報表管理數據庫時的錯誤

解決方案: 拷貝mysql驅動到cm的相關lib目錄下.

cp mysql-connector-java-5.1.31-bin.jar/usr/share/cmf/lib/

注:mysql驅動事先下載好.

f) 添加Hive服務指定mysql作為數據源時報找不到驅動的錯誤

解決方法:拷貝mysql驅動

cp mysql-connector-java-5.1.31-bin.jar/usr/share/java/

mv /usr/share/java/mysql-connector-java.jar

g) CM集群驗證錯誤

解決方法:按照提示修改告警

vi/etc/sysctl.conf

vm.swappiness= 0

/sbin/sysctl-p /etc/sysctl.conf

#確認是否修改為0

cat/proc/sys/vm/swappiness

#永久修復該告警

vi/etc/rc.local

#追加如下內容到上述文件內
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

#臨時修復該告警

echonever > /sys/kernel/mm/redhat_transparent_hugepage/defrag

4 配置

暫無

5 使用

登錄CM安裝IP和7180對CDH進行管理和維護.

6 卸載

暫無

7 總結

通過日志分析安裝的問題.

總結

以上是生活随笔為你收集整理的CDH Hadoop 基于CM方式半在线安装步骤详解的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。

上一篇：戴尔笔记本灵越15怎么进入Bios模式
下一篇： Apache Sentry手动安装、使用