1、關于版本問題：

hive-1.2.1+tez-0.9.0+hadoop-2.7.7
hive-2.3.6+tez-0.9.0+hadoop-2.7.1
兩種搭配均試過可行，之前配置的時候覺得與版本有很大關系，試過之后發現版本影響不大，還沒發現有問題

2、Tez安裝與配置

1、解壓并修改文件夾名字

[hadoop@hadoop01 tez]$ tar -zxvf apache-tez-0.9.0-bin.tar.gz [hadoop@hadoop01 tez]$ mv apache-tez-0.9.0-bin tez-0.9.0

2、在hdfs上創建/tez-0.9.0目錄，并tez.tar.gz文件上傳至此目錄

[hadoop@hadoop01 hadoop]$ hdfs dfs -mkdir /tez-0.9.0 [hadoop@hadoop01 hadoop]$ hdfs dfs -put /home/hadoop/app/tez/tez-0.9.0/share/tez.tar.gz /tez-0.9.0

3、更換tez下的lib目錄中的hadoop包的版本，將其與hadoop版本一致。
刪除/tez-0.9.0/lib下的hadoop包，并復制hadoop包至此文件夾

[hadoop@hadoop01 lib]# rm hadoop-mapreduce-client-core-2.7.0.jar hadoop-mapreduce-client-common-2.7.0.jar cp /home/hadoop/app/hdfs/hadoop/hadoop-2.7.7/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.7.jar hadoop-mapreduce-client-core-2.7.7.jar ./

4、在hadoop/hadoop-2.7.7/etc/hadoop/目錄 下增加 tez-site.xml文件

[hadoop@hadoop01 hadoop]$ vi tez-site.xml <?xml version="1.0" encoding="UTF-8"?> <configuration> <property> <name>tez.lib.uris</name> <value>${fs.defaultFS}/tez-0.9.0/tez.tar.gz</value> </property><property> <name>tez.container.max.java.heap.fraction</name> <value>0.2</value> </property><property><name>tez.use.cluster.hadoop-libs</name><value>true</value></property><property><name>tez.history.logging.service.class</name><value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value></property></configuration>

并將將文件發送至其他節點

[hadoop@hadoop02 hadoop]$ scp tez-site.xml hadoop@hadoop02:/home/hadoop/app/hdfs/hadoop/hadoop-2.7.7/etc/hadoop [hadoop@hadoop02 hadoop]$ scp tez-site.xml hadoop@hadoop03:/home/hadoop/app/hdfs/hadoop/hadoop-2.7.7/etc/hadoop

3、配置tez至hive上

1、在hive-env-sh 新增以下內容

export TEZ_HOME=/home/hadoop/app/tez/tez-0.9.0 export TEZ_JARS="" for jar in `ls $TEZ_HOME |grep jar`; doexport TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar done for jar in `ls $TEZ_HOME/lib`; doexport TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar done #在hadoop下找對應的lzo包 export HIVE_AUX_JARS_PATH=$HADOOP_HOME/share/hadoop/common/hadoop-lzo-0.4.21-SNAPSHOT.jar$TEZ_JARS

2、啟動hadoop與hive并測試

hive> set hive.execution.engine=tez; hive> select> age,> count(*)> from tz> group by age> ; Query ID = hadoop_20190920100059_baf244cb-d9ce-435b-a74a-f884a0d95835 Total jobs = 1 Launching Job 1 out of 1Status: Running (Executing on YARN cluster with App id application_1568944648748_0001)--------------------------------------------------------------------------------VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED -------------------------------------------------------------------------------- Map 1 .......... SUCCEEDED 1 1 0 0 0 0 Reducer 2 ...... SUCCEEDED 1 1 0 0 0 0 -------------------------------------------------------------------------------- VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 26.91 s -------------------------------------------------------------------------------- OK 16 2 18 3 Time taken: 64.555 seconds, Fetched: 2 row(s)

成功。

注：關于在hadoop集群所有在yarn上走的mr均使用tez配置

安裝Tez在Hadoop上會對集群影響，使得所有在yarn上運行的mapreduce都只能走tez引擎。

1、修改hadoop-env.sh增加以下內容

export TEZ_HOME=/opt/moudle/tez #是你的tez的解壓安裝目錄 for jar in `ls $TEZ_HOME |grep jar`; doexport HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$TEZ_HOME/$jar done for jar in `ls $TEZ_HOME/lib`; doexport HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$TEZ_HOME/lib/$jar done

2、修改mapred-site.xml

<property><name>mapreduce.framework.name</name><value>yarn-tez</value> </property>

3、將文件同步至集群其他節點

總結

以上是生活随笔為你收集整理的Hive更换TEZ引擎的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。