Hortworks Hadoop 2.4.2安装、配置
Hortonworks?Hadoop 2.4.2安裝、配置
?
1簡介
此手冊應用于Hortonworks Hadoop(HDP2.4.2)的安裝、使用。
1.1? 參考資料
http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Installing_HDP_AMB/bk_Installing_HDP_AMB-20160509.pdf
http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_ambari_reference_guide/bk_ambari_reference_guide-20160509.pdf
1.2? 環境說明
操作系統 64位redhat6.5,JDK 1.7.67,mysql 5.6.14
2? 常規hortonworks官方文檔安裝步驟
2.1? Ambari、HDP兼容關系表
2.2??? 基本配置要求
操作系統：
? Red Hat Enterprise Linux (RHEL) v7.x
? Red Hat Enterprise Linux (RHEL) v6.x
? CentOS v7.x
? CentOS v6.x
? Debian v7.x
? Oracle Linux v7.x
? Oracle Linux v6.x
? SUSE Linux Enterprise Server (SLES) v11 SP4 (HDP 2.2 and later)
? SUSE Linux Enterprise Server (SLES) v11 SP3
? SUSE Linux Enterprise Server (SLES) v11 SP1 (HDP 2.2)
? Ubuntu Precise v12.04
? Ubuntu Trusty v14.04
瀏覽器:
Windows (7, 8)
? Internet Explorer 10
? Firefox 18
? Google Chrome 26
? Mac OS X (10.6 or later)
? Firefox 18
? Safari 5
? Google Chrome 26
? Linux (CentOS, Debian, Oracle Linux, RHEL, SLES, Ubuntu)
? Firefox 18
? Google Chrome 26
軟件：
yum and rpm (RHEL/CentOS/Oracle Linux)
? zypper and php_curl (SLES)
? apt (Debian/Ubuntu)
? scp, curl, unzip, tar, and wget
? OpenSSL (v1.01, build 16 or later)
? Python
? For CentOS 6: Python 2.6.*
? For SLES 11: Python 2.6.8 or later
? For CentOS 7, Ubuntu 12,Ubuntu 14, and Debian 7: Python 2.7.*
JDK：
? Oracle JDK 1.8 64-bit (minimum JDK 1.8_60) (default)
? Oracle JDK 1.7 64-bit (minimum JDK 1.7_67)
? OpenJDK 8 64-bit (not supported on SLES)
? OpenJDK 7 64-bit (not supported on SLES)
元數據存儲：（Postgres 8.x, 9.3+、MySQL 5.6、Oracle 11g r2、SQL Server 2008 R2+）
? PostgreSQL 8
? PostgreSQL 9.1.13+, 9.3
? MySQL 5.6
? Oracle 11gr2, 12c
注:這里推薦以mysql作為元數據數據庫
內存要求：
Ambari所在機器至少有1 GB RAM,其中有500M空閑.
2.3? 安裝規劃
? 每臺節點里配置主機全域名,The fully qualifieddomain name (FQDN)
? 每個主機里要安裝組件規劃
? 要存儲數據的主目錄
? NameNode數據目錄
? DataNodes數據目錄
? Secondary NameNode數據目錄
? Oozie數據目錄
? YARN數據目錄
? ZooKeeper數據目錄
? 各種日志、pid、數據庫文件目錄
2.4? 準備環境
注:以下以Redhat 6.5為例.
2.4.1?? 配置SSH免密碼登錄
#主節點里執行如下步驟
ssh-keygen
cd ~/.ssh/
cat id_rsa.pub >>authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
#分別拷貝節點內的id_rsa.pub到一臺主節點內,并追加到主節點的authorized_keys
#分發主節點里配置好的authorized_keys到各從節點
2.4.2???開啟NTP服務
yum install ntpd
chkconfig ntpd ?開啟
注: 默認ntpd服務已經安裝,如果沒安裝,執行yum install ntpd(配置本地源的的方式或則連外網源)
2.4.3???檢查DNS和NSCD
以ambari在安裝時需要配置全域名，所以需要檢查DNS。為了減輕DNS的負擔, 建議在節點里用 Name Service Caching Daemon (NSCD)。
vi /etc/hosts
192.168.56.21name.hadoop
192.168.56.22datanode1.hadoop
192.168.56.23datanode2.hadoop
#每臺節點里配置FQDN，如下以主節點為例.
vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=name.hadoop
2.4.4?? 關閉防火墻
chkconfig iptables off
/etc/init.d/iptables stop
#service iptables stop
2.4.5?? 關閉SELinux
1）查看SELinux狀態：
1、/usr/sbin/sestatus -v
##如果SELinux status參數為enabled即為開啟狀態
SELinux status: ??enabled
2、getenforce ?##也可以用這個命令檢查
2）關閉SELinux：
1、臨時關閉（不用重啟機器）：
setenforce 0 ?##設置SELinux 成為permissive模式
##setenforce 1 設置SELinux 成為enforcing模式
2、修改配置文件需要重啟機器：
vi /etc/selinux/config
將SELINUX=enforcing改為SELINUX=disabled
3）重啟機器即可
2.4.6?? 創建系統用戶和系統組
添加ambari安裝、運行用戶到組.
useradd -G amabri amabri
2.4.7??配置操作系統本地源
1） 虛擬機里加載操作系統iso鏡像.
#選中一虛擬機點擊設置→點擊存儲→點擊沒有盤片→選擇ISO所在路徑。
2) 創建目錄以掛載操作系統鏡像
mkdir /mnt/cdrom? ?
3) 再掛載鏡像到新建目錄下
mount -t iso9660 /dev/`lsblk |grep rom | awk '{print $1}'` /mnt/cdrom
4) 配置本地系統源
cd /etc/yum.repos.d
cp rhel-source.repo rhel.repo
mv rhel-source.repo source.repo_bak
vi rhel.repo
[rhel-source]
name=RedHat Enterprise Linux $releasever - $basearch - Source
baseurl=file:///mnt/cdrom/
enabled=1
gpgcheck=1
gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-redhat-release
yum clean all
yum list update
yum makecache
2.4.8??安裝JDK
#安裝前,需要卸載系統內置的JDK
rpm -qa | grep jdk | xargs rpm -e--nodeps
#安裝解壓版jdk
cd /mnt/sf/public
tar –zxvf jdk-7u67-linux-x64.tar.gz
vi /etc/profile
exportJAVA_HOME=/mnt/sf/public/jdk1.7.0_67
exportCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
2.4.9??安裝http服務器
yum install httpd
service httpd start
#修改網站默認的跟目錄
vi /etc/httpd/conf/httpd.conf
#
# DocumentRoot: Thedirectory out of which you will serve your
# documents. Bydefault, all requests are taken from this directory, but
# symbolic links andaliases may be used to point to other locations.
#
DocumentRoot"/mnt/sf/html"
#
# This should bechanged to whatever you set DocumentRoot to.
#
<Directory"/mnt/sf/html">
2.4.10?安裝mysql
rpm –ivh MySQL-server-5.6.14-1.el6.x86_64.rpm
rpm –ivh MySQL-client-5.6.14-1.el6.x86_64.rpm
service mysql start
cat /root/.mysql_secret
#The random password set for the root user at Fri May 20 17:54:33 2016 (localtime): aX71HiTT
mysql –uroot -p aX71HiTT
set password=password(‘root’)
#創建用戶并分配權限
CREATE USER 'ambari'@'%'IDENTIFIED BY 'ambari';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
GRANT ALL PRIVILEGES ON *.* TO root@'%';
FLUSH PRIVILEGES;
2.4.11?配置安裝源
注:鑒于多數集群生產環境不能上網,所以安裝時采用本地源的方式.具體為先下載附錄中的ambari、HDP、HDP-UTILS的tar包然后再配置到本地源中。
#ambari和HDP(含HDP-UTILS)源
a)Ambari2.2.2源
| 操作系統 | 格式 | URL地址 |
| RedHat 6 CentOS 6 Oracle Linux 6 | Base URL | http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0 |
| Repo File | http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0/ambari.repo | |
| Tarball md5 | asc | http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0/ ambari-2.2.2.0-centos6.tar.gz |
b)HDP 2.4源
| 操作系統 | 版本號 | 源名稱 | 格式 | URL地址 |
| RedHat 6 CentOS 6 Oracle Linux 6 | HDP-2.4.2.0 | HDP | Base URL | http://public-repo-1.hortonworks.com/HDP/centos6/2.x/ updates/2.4.2.0 |
| Repo File | http://public-repo-1.hortonworks.com/HDP/centos6/2.x/ updates/2.4.2.0/hdp.repo | |||
| Tarball md5 | asc | http://public-repo-1.hortonworks.com/HDP/centos6/2.x/ updates/2.4.2.0/HDP-2.4.2.0-centos6-rpm.tar.gz | |||
| HDP-UTILS | Base URL | http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/ repos/centos6 | ||
| Repo File | http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/ repos/centos6/HDP-UTILS-1.1.0.20-centos6.tar.gz |
1)? 下載如下組件并拷貝到主節點的 httpd網站根目錄,即/mnt/sf/html內后解壓
Ambari 2.2.2:
http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0/ambari-2.2.2.0-centos6.tar.gz
HDP 2.4.2:
http://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/centos6/2.x/2.4-latest/HDP-2.4-latest-centos6-rpm.tar.gz
HDP-UTILS-1.1.0.20：
http://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6/HDP-UTILS-1.1.0.20-centos6.tar.gz
2)? 配置ambari、HDP、HDP-UTILS的源
注:事實證明HDP、HDP-UTILS源不需要配置，ambari安裝時可以指定本地源,并在各個節點里創建repo文件.ambari的源需要配置，因為安裝采用yum install ambari方式.
vi ambari.repo
#VERSION_NUMBER=2.2.2.0-460
[Updates-ambari-2.2.2.0]
name=ambari-2.2.2.0 -Updates
baseurl=http://192.168.56.21/AMBARI-2.2.2.0/centos6/2.2.2.0-460/
gpgcheck=1
gpgkey=http://192.168.56.21/AMBARI-2.2.2.0/centos6/2.2.2.0-460/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
vi hdp.repo
#VERSION_NUMBER=2.4.2.0-258
[HDP-2.4.2.0]
name=HDP Version -HDP-2.4.2.0
baseurl=http://192.168.56.21/HDP/centos6/2.x/updates/2.4.2.0
gpgcheck=1
gpgkey=http://192.168.56.21/HDP/centos6/2.x/updates/2.4.2.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
?
[HDP-UTILS-1.1.0.20]
name=HDP Utils Version- HDP-UTILS-1.1.0.20
baseurl=http://192.168.56.21/HDP-UTILS-1.1.0.20/repos/centos6
gpgcheck=1
gpgkey=http://192.168.56.21/HDP/centos6/2.x/updates/2.4.2.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
yum clean all
yum list update
yum makecache
yum repolist
2.5? 執行安裝
Yum方式安裝ambari。
2.5.1??安裝ambari2.2.2
1）安裝ambari
yum install ambari-server
2）配置amabri
ambari-server setup
Customize user account for ambari-server daemon [y/n] (n)?
輸入：y
Enter user account for ambari-server daemon (root):
注:這里亦可選擇用戶amabri,屆時會讓提供該用戶的密碼等信息.
輸入：root
檢查防火墻是否關閉
Adjusting ambari-server permissions and ownership...
Checking firewall...
WARNING: iptables is running. Confirm the necessary Ambari ports areaccessible. Refer to the Ambari documentation for more details on ports.
OK to continue [y/n] (y)?
輸入：y
檢查JDK
Checking JDK...
[1] - Oracle JDK 1.7 + Java Cryptography Extension (JCE)Policy Files 7
[2] - Oracle JDK 1.6 + Java Cryptography Extension (JCE) Policy Files 6
[3] - Custom JDK
==============================================================================
Enter choice (1):
輸入：3
輸入：/mnt/sf/public/jdk1.7.0_67
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL
[4] - PostgreSQL
==============================================================================
Enter choice (1):
輸入：3
Hostname (localhost):
Port (3306):
Database Name (ambari):
Username (ambari):
輸入：（什么也不輸入，直接“回車”）
Enter Database Password (cluster):
Re-enter password:
輸入：ambari
WARNING: Before starting Ambari Server, youmust run the following DDL against the database to create the schema:/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
注:這里需要以mysql的amabri用戶在ambari數據庫里source上面的sql文件，作用時初始化ambari數據庫。
Proceed with configuring remote database connectionproperties [y/n] (y)?
輸入：y
2） 啟動ambari服務
ambari-server start
Using python??/usr/bin/python2.6
Starting ambari-server
Ambari Server running with 'root' privileges.
Organizing resource files at/var/lib/ambari-server/resources...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start....................
Ambari Server 'start' completed successfully.
注:啟動ambari訪問前，請確保mysql驅動已經放置在/usr/share/java內且名字是mysql-connector-java.jar不然ambari server啟動時會報錯mysql驅動找不到的錯誤.具體報錯見:
21 May 2016 20:48:28,985 ERROR [main]DBAccessorImpl:106 - Error while creating database accessor
java.lang.ClassNotFoundException:com.mysql.jdbc.Driver
atjava.net.URLClassLoader$1.run(URLClassLoader.java:366)
atjava.net.URLClassLoader$1.run(URLClassLoader.java:355)
atjava.security.AccessController.doPrivileged(Native Method)
atjava.net.URLClassLoader.findClass(URLClassLoader.java:354)
atjava.lang.ClassLoader.loadClass(ClassLoader.java:425)
atsun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
atjava.lang.ClassLoader.loadClass(ClassLoader.java:358)
atjava.lang.Class.forName0(Native Method)
atjava.lang.Class.forName(Class.java:190)
2.5.2??安裝HDP2.4.2
登錄ambari服務器以8080端口訪問.如下是ambari界面化安裝時的部分截圖:
#集群初始化
?
?
注:主機確認有個openssl的bug導致主機注冊始終不通過。具體報錯見:
Failed to connect to https://name.hadoop:8440/cert/cadue to [Errno 1] _ssl.c:492: error:100AE081:elliptic curveroutines:EC_GROUP_new_by_curve_name:unknown group
解決辦法是升級openssl-1.0.1e-15.el6.x86_64到更高版本,具體做法是配置163操作系統源.
在http://mirrors.163.com/.help/centos.html里點擊下載centos6的源
拷貝該repo文件到/etc/ yum.repos.d
#編輯該源文件并修改內容成如下:
vi CentOS6-Base-163.repo
[base]
name=CentOS-$releasever - Base - 163.com
baseurl=http://mirrors.163.com/centos/6/os/x86_64/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
gpgcheck=0
#gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
yum clean all
yum list update
yum makecache
yum upgrade openssl
#分發到其它節點做同樣的事情
scp CentOS6-Base-163.reporoot@192.168.56.22:/etc/yum.repos.d/
主機檢查時出現TransparentHuge Pages Issues (1)
解決方法:執行下面的命令可臨時解決這個警告：
echo never >/sys/kernel/mm/redhat_transparent_hugepage/defrag
永久解決：
vi? /etc/rc.local
在文件的最后添加下面的行，保存即可。
echo never >/sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never >/sys/kernel/mm/transparent_hugepage/enabled
前期配置順利，直接按照界面化安裝出錯信息，排錯。很容易到最終的安裝.
3? 問題
已分散到各章節內。
4總結
具體問題具體分析，以log為準。
與50位技術專家面對面20年技術見證，附贈技術全景圖總結
以上是生活随笔為你收集整理的Hortworks Hadoop 2.4.2安装、配置的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： Virtaulbox虚拟机添加磁盘
- 下一篇： Hive多字符列分割