當(dāng)前位置：首頁(yè) > 编程资源 > 编程问答 >内容正文

编程问答

CHD6.3.1部署文档

發(fā)布時(shí)間：2023/12/18 编程问答 17 豆豆

生活随笔收集整理的這篇文章主要介紹了 CHD6.3.1部署文档小編覺(jué)得挺不錯(cuò)的,現(xiàn)在分享給大家,幫大家做個(gè)參考.

文章目錄

- MySQL離線部署
- CHD分為兩塊
機(jī)器準(zhǔn)備
安裝包準(zhǔn)備
- - MySql離線部署
  - CM
  - Parcel文件
  - JDK
集群節(jié)點(diǎn)初始化
- - 1 修改hosts文件
  - 2 關(guān)閉防火墻&清空規(guī)則
  - 3 關(guān)閉所有節(jié)點(diǎn)的selinux
  - 4 統(tǒng)一時(shí)間
  - - - 4-1 主從
      - 4-2 主主
      - 4-3 公司內(nèi)部時(shí)鐘服務(wù)器 maven倉(cāng)庫(kù)中心
  - 5 JDK
  - 6 離線部署MySQL
  - 7 創(chuàng)建CDH元數(shù)據(jù)的用戶和amon的服務(wù)的庫(kù)
  - - - 選擇第一臺(tái)部署 amon進(jìn)程那么mysql jar包就需要
    - 11 部署CM Server+agent rpm
  - 需要安裝的模塊
  - - Server:
    - agent:

MySQL離線部署

CHD分為兩塊

CM 不開(kāi)源主從架構(gòu) server + 多個(gè)agent 離線部署
Parcel 包裹文件：就是

將Hadoop / Hive等壓縮包完善之后壓縮后重命名parcel結(jié)尾用 tar -xzvf xxx.parcel 也可以解壓

機(jī)器準(zhǔn)備

安裝包準(zhǔn)備

MySql離線部署

mysql-5.7.11-linux-glibc2.5-x86_64.tat.gz
mysql-connector-java-5.1.47.jar

CM

cm-6.3.1-radhat7.tar.gz

Parcel文件

CDH-6.3.1-1.cdh6.3.1.p0.1470567-e17.parcel
CDH-6.3.1-1.cdh6.3.1.p0.1470567-e17.parcel.shal
maifest.json

JDK

jdk-8u45-linux-x64.gz

集群節(jié)點(diǎn)初始化

1 修改hosts文件

首先工程師PC的hosts文件

MAC: /etc/hosts
Window: C:\windows\system32\drivers\etc\hosts

公?地址機(jī)器名稱: 47.74.130.251 ruozedata001 161.117.1.169 ruozedata002 47.88.170.121 ruozedata003

第二修改服務(wù)器防止ip改變之后造成的大工作量的更改

echo '"172.21.230.219 ifeng001" >> /ect/hosts echo '"172.21.230.220 ifeng002" >> /ect/hosts echo '"172.21.230.221 ifeng003" >> /ect/hosts

2 關(guān)閉防火墻&清空規(guī)則

#停止 systemctl stop firewalled #禁用 systemctl disable firewalld#清空防火墻的規(guī)則 iptables -F

3 關(guān)閉所有節(jié)點(diǎn)的selinux

#安全增強(qiáng)型 Linux（Security-Enhanced Linux）簡(jiǎn)稱 SELinux vi /etc/selinux/config

4 統(tǒng)一時(shí)間

# 查看日期 date# 設(shè)置時(shí)區(qū) timedatectl --help

4-1 主從

主和互聯(lián)網(wǎng)同步，從節(jié)點(diǎn)與主節(jié)點(diǎn)同步

4-2 主主

每一臺(tái)都與互聯(lián)網(wǎng)同步

4-3 公司內(nèi)部時(shí)鐘服務(wù)器 maven倉(cāng)庫(kù)中心

#部署ntp yum install -y ntp # 選擇第一臺(tái)機(jī)器作為主節(jié)點(diǎn) vi /etc/ntp.conf

當(dāng)網(wǎng)絡(luò)時(shí)間不能用，用本地時(shí)間

restrict 控制相關(guān)權(quán)限。語(yǔ)法為： restrict IP地址 mask 子網(wǎng)掩碼參數(shù)其中IP地址也可以是default ，default 就是指所有的IP參數(shù)有以下幾個(gè)：ignore ：關(guān)閉所有的 NTP 聯(lián)機(jī)服務(wù)nomodify：客戶端不能更改服務(wù)端的時(shí)間參數(shù)，但是客戶端可以通過(guò)服務(wù)端進(jìn)行網(wǎng)絡(luò)校時(shí)。notrust ：客戶端除非通過(guò)認(rèn)證，否則該客戶端來(lái)源將被視為不信任子網(wǎng)noquery ：不提供客戶端的時(shí)間查詢：用戶端不能使用ntpq，ntpc等命令來(lái)查詢ntp服務(wù)器notrap ：不提供trap遠(yuǎn)端登陸：拒絕為匹配的主機(jī)提供模式 6 控制消息陷阱服務(wù)。陷阱服務(wù)是 ntpdq 控制消息協(xié)議的子系統(tǒng)，用于遠(yuǎn)程事件日志記錄程序。nopeer ：用于阻止主機(jī)嘗試與服務(wù)器對(duì)等，并允許欺詐性服務(wù)器控制時(shí)鐘kod ：訪問(wèn)違規(guī)時(shí)發(fā)送 KoD 包。restrict -6 表示IPV6地址的權(quán)限設(shè)置 # 主節(jié)點(diǎn)只接受172.21.230的網(wǎng)段鏈接 restrict 172.21.230.0 mask 255.255.255.0 nomodify notrap # 重新啟動(dòng)服務(wù) systemctl start

其他節(jié)點(diǎn)與主節(jié)點(diǎn)鏈接

# sotp systemctl stop ntpd# 禁止ntpd啟動(dòng) systemctl disable ntpdntpdate ifeng001 # 每天同步一次 crontab -e * * * * * /usr/sbin/ntpdate ifeng001

5 JDK

5.1 傳送兩個(gè)文件

scp -r CDH6.3.1 ifeng002:/root/ mkdir CDH6.3.1 mkdir /usr/java/tar -xzvf jdk-8u45-linux-x64.gz -C /usr/java/

修正用戶和用戶組

chown -R root:root /usr/java/jdk1.8.0_45echo "export JAVA_HOME=/usr/java/jdk1.8.0_45" >> /etc/profile echo "export PATH={JAVA_HOME}/bin:${PATH}" >> /etc/ source /etc/profilewhich java

6 離線部署MySQL

1.解壓及創(chuàng)建目錄

[root@hadoop39 local]# tar xzvf mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz -C [root@hadoop39 local]# mv mysql-5.7.11-linux-glibc2.5-x86_64 mysql[root@hadoop39 local]# mkdir mysql/arch mysql/data mysql/tmp

2.創(chuàng)建my.cnf(見(jiàn)文件)

[root@hadoop39 local]# vi /etc/my.cnf [client] port = 3306 socket = /usr/local/mysql/data/mysql.sock default-character-set=utf8mb4[mysqld] port = 3306 socket = /usr/local/mysql/data/mysql.sockskip-slave-startskip-external-locking key_buffer_size = 256M sort_buffer_size = 2M read_buffer_size = 2M read_rnd_buffer_size = 4M query_cache_size= 32M max_allowed_packet = 16M myisam_sort_buffer_size=128M tmp_table_size=32Mtable_open_cache = 512 thread_cache_size = 8 wait_timeout = 86400 interactive_timeout = 86400 max_connections = 600# Try number of CPU's*2 for thread_concurrency #thread_concurrency = 32 #isolation level and default engine default-storage-engine = INNODB transaction-isolation = READ-COMMITTEDserver-id = 1739 basedir = /usr/local/mysql datadir = /usr/local/mysql/data pid-file = /usr/local/mysql/data/hostname.pid#open performance schema log-warnings sysdate-is-nowbinlog_format = ROW log_bin_trust_function_creators=1 log-error = /usr/local/mysql/data/hostname.err log-bin = /usr/local/mysql/arch/mysql-bin expire_logs_days = 7innodb_write_io_threads=16relay-log = /usr/local/mysql/relay_log/relay-log relay-log-index = /usr/local/mysql/relay_log/relay-log.index relay_log_info_file= /usr/local/mysql/relay_log/relay-log.infolog_slave_updates=1 gtid_mode=OFF enforce_gtid_consistency=OFF# slave slave-parallel-type=LOGICAL_CLOCK slave-parallel-workers=4 master_info_repository=TABLE relay_log_info_repository=TABLE relay_log_recovery=ON#other logs #general_log =1 #general_log_file = /usr/local/mysql/data/general_log.err #slow_query_log=1 #slow_query_log_file=/usr/local/mysql/data/slow_log.err#for replication slave sync_binlog = 500#for innodb options innodb_data_home_dir = /usr/local/mysql/data/ innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextendinnodb_log_group_home_dir = /usr/local/mysql/arch innodb_log_files_in_group = 4 innodb_log_file_size = 1G innodb_log_buffer_size = 200M#根據(jù)生產(chǎn)需要，調(diào)整pool size innodb_buffer_pool_size = 12G #innodb_additional_mem_pool_size = 50M #deprecated in 5.6 tmpdir = /usr/local/mysql/tmpinnodb_lock_wait_timeout = 1000 #innodb_thread_concurrency = 0 innodb_flush_log_at_trx_commit = 2innodb_locks_unsafe_for_binlog=1#innodb io features: add for mysql5.5.8 performance_schema innodb_read_io_threads=4 innodb-write-io-threads=4 innodb-io-capacity=200 #purge threads change default(0) to 1 for purge innodb_purge_threads=1 innodb_use_native_aio=on#case-sensitive file names and separate tablespace innodb_file_per_table = 1 lower_case_table_names=1[mysqldump] quick max_allowed_packet = 128M[mysql] no-auto-rehash default-character-set=utf8mb4[mysqlhotcopy] interactive-timeout[myisamchk] key_buffer_size = 256M sort_buffer_size = 256M read_buffer = 2M write_buffer = 2M

3.創(chuàng)建用戶組及用戶

[root@hadoop39 local]# groupadd -g 101 dba [root@hadoop39 local]# useradd -u 514 -g dba -G root -d /usr/local/mysql mysqladmin [root@hadoop39 local]# id mysqladmin uid=514(mysqladmin) gid=101(dba) groups=101(dba),0(root)## 一般不需要設(shè)置mysqladmin的密碼，直接從root或者LDAP用戶sudo切換 #[root@hadoop39 local]# passwd mysqladmin Changing password for user mysqladmin. New UNIX password: BAD PASSWORD: it is too simplistic/systematic Retype new UNIX password: passwd: all authentication tokens updated successfully.## if user mysqladmin is existing,please execute the following command of usermod. #[root@hadoop39 local]# usermod -u 514 -g dba -G root -d /usr/local/mysql mysqladmin

4.copy 環(huán)境變量配置文件至mysqladmin用戶的home目錄中,為了以下步驟配置個(gè)人環(huán)境變量

[root@hadoop39 local]# cp /etc/skel/.* /usr/local/mysql ###important

5.配置環(huán)境變量

[root@hadoop39 local]# vi mysql/.bashrc # .bashrc # Get the aliases and functionsif [ -f ~/.bashrc ]; then. ~/.bashrc fi# User specific environment and startup programs export MYSQL_BASE=/usr/local/mysql export PATH=${MYSQL_BASE}/bin:$PATHunset USERNAME#stty erase ^H set umask to 022 umask 022 PS1=`uname -n`":"'$USER'":"'$PWD'":>"; export PS1## end

6.賦權(quán)限和用戶組，切換用戶mysqladmin，安裝

[root@hadoop39 local]# chown mysqladmin:dba /etc/my.cnf [root@hadoop39 local]# chmod 640 /etc/my.cnf [root@hadoop39 local]# chown -R mysqladmin:dba /usr/local/mysql [root@hadoop39 local]# chmod -R 755 /usr/local/mysql

7.配置服務(wù)及開(kāi)機(jī)自啟動(dòng)

[root@hadoop39 local]# cd /usr/local/mysql #將服務(wù)文件拷貝到init.d下，并重命名為mysql [root@hadoop39 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql #賦予可執(zhí)行權(quán)限 [root@hadoop39 mysql]# chmod +x /etc/rc.d/init.d/mysql #刪除服務(wù) [root@hadoop39 mysql]# chkconfig --del mysql #添加服務(wù) [root@hadoop39 mysql]# chkconfig --add mysql [root@hadoop39 mysql]# chkconfig --level 345 mysql on

8.安裝libaio及安裝mysql的初始db

[root@hadoop39 mysql]# yum -y install libaio [root@hadoop39 mysql]# sudo su - mysqladminhadoop39.ruoze:mysqladmin:/usr/local/mysql:> bin/mysqld \ --defaults-file=/etc/my.cnf \ --user=mysqladmin \ --basedir=/usr/local/mysql/ \ --datadir=/usr/local/mysql/data/ \ --initialize在初始化時(shí)如果加上 –initial-insecure，則會(huì)創(chuàng)建空密碼的 root@localhost 賬號(hào)，否則會(huì)創(chuàng)建帶密碼的 root@localhost 賬號(hào)，密碼直接寫在 log-error 日志文件中（在5.6版本中是放在 ~/.mysql_secret 文件里，更加隱蔽，不熟悉的話可能會(huì)無(wú)所適從）

9.查看臨時(shí)密碼

hadoop39.ruoze:mysqladmin:/usr/local/mysql/data:>cat hostname.err |grep password 2017-07-22T02:15:29.439671Z 1 [Note] A temporary password is generated for root@localhost: kFCqrXeh2y(0 hadoop39.ruoze:mysqladmin:/usr/local/mysql/data:>

10.啟動(dòng)

/usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf &

11.登錄及修改用戶密碼

hadoop39.ruoze:mysqladmin:/usr/local/mysql/data:>mysql -uroot -p'kFCqrXeh2y(0' mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.11-logCopyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> alter user root@localhost identified by 'ruozedata'; Query OK, 0 rows affected (0.05 sec)mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'ruozedata' ; Query OK, 0 rows affected, 1 warning (0.02 sec)mysql> flush privileges; Query OK, 0 rows affected (0.00 sec)mysql> exit; Bye

12.重啟

hadoop39.ruoze:mysqladmin:/usr/local/mysql:> service mysql restarthadoop39.ruoze:mysqladmin:/usr/local/mysql/data:>mysql -uroot -pruozedata mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.11-log MySQL Community Server (GPL)Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql>

7 創(chuàng)建CDH元數(shù)據(jù)的用戶和amon的服務(wù)的庫(kù)

create database cmf dafult chatacter ser utf8; create database amon dafult chatacter ser utf8;grant all privileges on cmf.* to 'cmf'@'%' identified by '123456' grant all privileges on amon.* to 'amon'@'%' identified by '123456'

選擇第一臺(tái)部署 amon進(jìn)程那么mysql jar包就需要

java目錄：/usr/java jar包目錄：/usr/share/java 創(chuàng)建jar包存放目錄 mkdir -p /usr/share/java重命名jar包 cp mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar

11 部署CM Server+agent rpm

tar -xzvf cm6.3.1-redhat7.tar.gz

tar開(kāi)之后進(jìn)入到文件夾cm6.3.1

需要安裝的模塊

Server:

cloudera-manager-daemons-6.3.1-1466458.e17.x86_64.rpm cloudera-manager-server-6.3.1-1466458.e17.x86_64.rpm先安裝核心的daemon

agent:

cloudera-manager-daemons-6.3.1-1466458.e17.x86_64.rpm cloudera-manager-agent-6.3.1-1466458.e17.x86_64.rpm先安裝核心的daemon rpm包

網(wǎng)絡(luò)允許的情況下：

yum install cloudera-manager-daemons-6.3.1-1466458.e17.x86_64.rpm

網(wǎng)絡(luò)不被放開(kāi)的情況下用

# 不去下載依賴包，直接裸裝這個(gè)rpm包 rpm -ivh cloudera-manager-daemons-6.3.1-1466458.e17.x86_64.rpm

總結(jié)

以上是生活随笔為你收集整理的CHD6.3.1部署文档的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。

文档

上一篇： [django]梳理drf知识点
下一篇：如何将火星坐标（高德、谷歌、腾讯地图坐标