Apache Ranger源码编译及使用
Ranger源碼編譯、使用手冊
1 Ranger簡介
Apache Ranger提供一個集中式安全管理框架,它可以對Hadoop生態的組件如Hive,Hbase進行細粒度的數據訪問控制.通過操作Ranger控制臺,管理員可以輕松的通過配置策略來控制用戶訪問HDFS文件夾、HDFS文件、數據庫、表、字段權限.這些策略可以為不同的用戶和組來設置,同時權限可與hadoop無縫對接.
2 準備
2.1 環境說明
1 Ranger源碼編譯依賴如下linux組件:maven,git,gcc,mysql
2 安裝git和gcc時采用yum安裝,請配置好本地源
3 安裝mysql是請確保清理好系統自帶的mysql.
2.1.1虛擬機里Linux系統版本
[root@localhost ranger-0.5.0-usersync]# cat /etc/issue | grep Linux
Red Hat Enterprise Linux Server release 6.5 (Santiago)
2.1.2 JDK版本
[root@localhost native]# java -version
java version "1.7.0_67"
注:官網強調一定是1.7以上版本.
Java(TM) SE RuntimeEnvironment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-BitServer VM (build 24.65-b04, mixed mode)
2.1.3mysql版本
[root@localhost native]# mysql -uroot -proot-e"select version()";
Warning: Using a password onthe command line interface can be insecure.
+-----------+
| version() |
+-----------+
| 5.6.14 ???|
+-----------+
注:
1 Mysql 驅動為mysql-connector-java-5.1.31-bin.jar
2 改jar被重命名后放置在/usr/share/java/內被其它Ranger插件共享
2.1.4 Maven版本
?[root@localhost bin]# mvn -version
Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9;2014-02-15T01:37:52+08:00)
Maven home: /root/maven-3.2.1
Java version: 1.7.0_67,vendor: Oracle Corporation
Java home:/root/jdk1.7.0_67/jre
Default locale: en_US,platform encoding: UTF-8
OS name: "linux",version: "2.6.32-431.el6.x86_64", arch: "amd64", family:"unix"
2.1.5 git版本
?[root@localhost native]# git version
git version 1.7.1
2.2 編譯準備
2.2.1 安裝maven
[root@localhost ~]# cd /root
#下載地址
#https://maven.apache.org/download.cgi 最新版
#http://apache.opencas.org/maven/binaries/apache-maven-3.2.1-bin.tar.gz
tar –zxvf apache-maven-3.2.1-bin.tar.gz
mv apache-maven-3.2.1-bin maven-3.2.1
#修改環境變量,在~/.bash_profile里定義MAVEN_HOME并追加到PATH里
export MAVEN_HOME=/root/maven-3.2.1
:$MAVEN_HOME/bin:$PATH
#source環境變量,測試maven版本
source ~/.bash_profile
mvn –version
2.2.2 安裝git
這里通過本地源yum方式安裝.
yum install git
注:Linux本地源配置見下:
1) 虛擬機加載Linux ISO鏡像
2) 找到rom對應設備名
[root@localhost ~]# lsblk
NAME??????????????????????? MAJ:MIN RM? SIZE RO TYPE MOUNTPOINT
sr0????????????????????????? 11:0??? 1 1024M?0 rom?
sr1????????????????????????? 11:1??? 1?3.6G? 0 rom?
sr2????????????????????????? 11:2??? 1 1024M?0 rom?
sda?????????????????????????? 8:0??? 0???8G? 0 disk
?..sda1???????????????????????8:1??? 0?500M? 0 part /boot
?..sda2??????????????????????? 8:2??? 0?7.5G? 0 part
??..VolGroup-lv_root (dm-0) 253:0???0? 6.7G? 0 lvm?/
??..VolGroup-lv_swap (dm-1) 253:1???0? 816M? 0 lvm?[SWAP]
3) 創建目錄
[root@localhost ~]#mkdir –p /mnt/cdrom/
4) 掛載鏡像
mount -t iso9660 /dev/sr1?/mnt/cdrom
5) 配置linux更新源,/etc/yum.repos.d,修改成如下:
vi /etc/yum.repos.d/redhat.repo
[rhel-source]
name=Redhat
baseurl=file:///mnt/cdrom/
enabled=1
gpgcheck=1
gpgkey=file:///mnt/cdrom//RPM-GPG-KEY-redhat-release
6) 更新更新源
yum clean all
yum update list
2.2.3 安裝gcc
yum install gcc
2.2.4 安裝mysql
1) 安裝Mysql服務、客戶端
rpm –ivh MySQL-shared-5.6.14-1.el6.x86_64.rpm
rpm –ivh MySQL-shared-compat-5.6.14-1.el6.x86_64.rpm
rpm –ivh MySQL-server-5.6.14-1.el6.x86_64.rpm
rpm –ivh MySQL-client-5.6.14-1.el6.x86_64.rpm
2) 啟動mysql服務
service mysql start
3) 修改mysql初始密碼,先找到安裝時的初始密碼,在修改成自己的密碼
[root@localhost ~]#cat /root/.mysql_secret
# The random password set for the root user at Tue Dec 2221:17:22 2015 (local time):RUmKBqcY
mysql –uroot -p RUmKBqcY
set password=password(‘root’)
3 編譯
3.1編譯中
1) 拷貝ranger源代碼
[root@localhost ~]# cd ~
git clone https://github.com/apache/incubator-ranger.git
cd?incubator-ranger
git checkout ranger-0.5
2) 編譯ranger源代碼
cd?~/incubator-ranger
export?MAVEN_OPTS="-Xmx512M"?
export?JAVA_HOME= /root/jdk1.7.0_67
export?PATH=$JAVA_HOME/bin:$PATH
mvn clean compile package assembly:assembly?install
ls?target/*.tar.gz
[root@localhost ~]#ls/root/incubator-ranger/target/*.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-admin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-hbase-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-hdfs-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-hive-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-kafka-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-kms.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-knox-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-migration-util.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-solr-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-src.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-storm-plugin.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-usersync.tar.gz
/root/incubator-ranger/target/ranger-0.5.0-yarn-plugin.tar.gz
3.2 編譯問題
1) 編譯過程異常緩慢,一般要3-4天時間
2) 如果出現異常不好定位,可在maven參數里加-X以debug模式診斷
3) 源碼編譯的相關問題及解決方案
a) Failedto execute goal on project ranger-hdfs-plugin: Could not resolve dependenciesfor project
???? security_plugins.ranger-hdfs-plugin:
???? ranger-hdfs-plugin:jar:0.5.0:The following artifacts could not be resolved:
org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde,eigenbase:eigenbase-properties:jar:1.1.4,net.hydromatic:linq4j:jar:0.4,net.hydromatic:quidem:jar:0.1.1:
Could not transfer artifactorg.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde from/to conjars(http://conjars.org/repo): conjars.org:Unknownhost conjars.org -> [Help 1]
?
解決方案:手動下載錯誤提示里的jar并拷貝到相應的m2目錄內.
源:http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.3-jhyde/
???? 目標:/root/.m2/repository/org/pentaho/pentaho-aggdesigner/5.1.3-jhyde
? 源:http://conjars.org/repo/eigenbase/eigenbase-properties/1.1.4/
? 目標:/root/.m2/repository/eigenbase/eigenbase-properties/1.1.4/
? 源:http://conjars.org/repo/net/hydromatic/linq4j/0.4/
? 目標:/root/.m2/repository/net/hydromatic/linq4j/0.4/
? 源:http://conjars.org/repo/net/hydromatic/quidem/0.1.1/
? 目標:/root/.m2/repository/net/hydromatic/quidem/0.1.1/
?
b)[ERROR]error: error reading /root/.m2/repository/org/json/json/20090211/json-20090211.jar;zip file is empty
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException:Failed to execute goalorg.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) onproject ranger-hdfs-plugin: Compilation failure
error: error reading/root/.m2/repository/org/json/json/20090211/json-20090211.jar; zip file isempty
?
???? atorg.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
.
解決方案:手動下載錯誤提示里的json-20090211.jar并拷貝到相應的m2目錄內.
c) Runningorg.apache.hadoop.crypto.key.kms.server.TestKeyAuthorizationKeyProvider此步耗時較長,大約20分鐘左右
4 配置
Ranger在solr里存儲日志,RangerAdmin UI依賴solr組件完成審計日志的查詢,所以需要先安裝和配置好Solr
注:目前(HDFS-Plugin)的測試日志審計時沒選擇Solr方式,但還是先配置好Standalone模式的solr.
4.1 Solr或SolrCloud安裝配置
cd /root/incubator-ranger/security-admin/contrib/solr_for_audit_setup
#打開install.properties文件,修改參數的值如下所示:
vi install.properties
JAVA_HOME=/root/jdk1.7.0_67
SOLR_INSTALL=true
SOLR_DOWNLOAD_URL=http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz
SOLR_INSTALL_FOLDER=/opt/solr
SOLR_RANGER_HOME=/opt/solr/ranger_audit_server
SOLR_DEPLOYMENT=standalone
SOLR_RANGER_DATA_FOLDER=/opt/solr/ranger_audit_server/data
SOLR_LOG_FOLDER=/var/log/solr/ranger_audits
SOLR_MAX_MEM=2g
#安裝單節點的solr
./ setup.sh
#按照如下安裝提示啟動、停止solr服務
cat/opt/solr/ranger_audit_server/install_notes.txt
#啟動solr
/opt/solr/ranger_audit_server/scripts/start_solr.sh
#驗證solr服務是否可正常使用
lsof –i:6083
http://192.168.56.101:6083
#確保防火墻已經關閉
chkconfig?iptables?off
4.2 Ranger Admin安裝配置
cp/root/incubator-ranger/target/ranger-0.5.0-admin.tar.gz /root
cd /root
tar –zxvf ranger-0.5.0-admin.tar.gz
cd ranger-0.5.0-admin
#打開Ranger Admin里install.properties文件,修改參數的值如下所示:
vi install.properties
setup_mode=SeparateDB
DB_FLAVOR=MYSQL
db_root_user=root
db_root_password=root
db_host=localhost
db_name=ranger
db_user=root
db_password=root
audit_store=db
audit_db_name=ranger_audit
audit_db_user=root
audit_db_password=root
policymgr_external_url=http://localhost:6080
policymgr_http_enabled=true
unix_user=ranger
unix_group=ranger
#安裝Ranger Admin
./setup.sh
#啟動Ranger Admin服務
ranger-admin start
#驗證Ranger Admin服務,如果出現Ranger的登錄界面,說Okay了.注:用戶名/密碼 admin/admin
lsof –i:6080
http://192.168.56.101:6080
4.3 Ranger-usersync安裝配置
cp/root/incubator-ranger/target/ranger-0.5.0-usersync.tar.gz /root/
cd /root
tar –zxvf ranger-0.5.0-usersync.tar.gz
cd ranger-0.5.0-usersync
#打開usersync Plugin里install.properties文件,修改參數的值如下所示:
vi install.properties
POLICY_MGR_URL=http://localhost:6080
SYNC_SOURCE=unix
#同步周期,1分鐘
SYNC_INTERVAL=1
logdir=/var/log/ranger/usersync
#安裝usersync Plugin
./setup.sh
#啟用usersync Plugin插件
./ranger--usersync-services.sh start
?
4.4 HDFS-Plugin安裝配置
cp/root/incubator-ranger/target/ranger-0.5.0-hdfs-plugin.tar.gz /root
cd /root
tar –zxvf ranger-0.5.0-hdfs-plugin.tar.gz
cd ranger-0.5.0-hdfs-plugin
#打開HDFS Plugin里install.properties文件,修改參數的值如下所示:
vi install.properties
POLICY_MGR_URL=http://localhost:6080
SQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar
REPOSITORY_NAME=hadoopdev
XAAUDIT.DB.IS_ENABLED=true
XAAUDIT.DB.FLAVOUR=MYSQL
XAAUDIT.DB.HOSTNAME=localhost
XAAUDIT.DB.DATABASE_NAME=ranger_audit
XAAUDIT.DB.USER_NAME=root
XAAUDIT.DB.PASSWORD=root
#組件對應的用戶,這里設置為空.一般Hadoop的內置用戶是HDFS或則hadoop
CUSTOM_USER=root
CUSTOM_GROUP=root
#啟用HDFS Plugin插件
[root@localhost ranger-0.5.0-hdfs-plugin]#./enable-hdfs-plugin.sh
Customuser and group are not available, using default user and group.
ERROR:Unable to find the conf directory of component [hadoop]; dir [/root/hadoop/conf] not found.
Exitinginstallation.
注:這里報錯,需要額外將HADOOP的conf做個軟連接到/root/hadoop/conf.
ln-s /root/hadoop-2.7.1/etc/hadoop/ /root/hadoop/conf
#再次啟用HDFS Plugin插件
[root@localhost ranger-0.5.0-hdfs-plugin]# ./enable-hdfs-plugin.sh
Custom user and group are not available,using default user and group.
ERROR: Unable to find the lib directory ofcomponent [hadoop];? dir [/root/hadoop/lib] not found.
Exiting installation.
#這里需要將HDFS Plugin內的jar和HADOOP包含的HDFS jar都指向/root/hadoop/lib
cp /root/ranger-0.5.0-hdfs-plugin/lib/ranger-hdfs-plugin-impl/*.jar/root/hadoop-2.7.1/share/hadoop/hdfs/lib/
mkdir /root/hadoop/lib
ln -s /root/hadoop-2.7.1/share/hadoop/hdfs/lib//root/hadoop/lib/
#再一次啟用HDFS Plugin插件
[root@localhost ranger-0.5.0-hdfs-plugin]# ./enable-hdfs-plugin.sh
#驗證HDFS Plugin服務,這時登麗Ranger的管理員界面驗證下HDFS plugin是夠加載成功,發現并沒有.
?
原因是安裝HDFS plugin時install.properties文件里定義的REPOSITORY_NAME(值為hadoopdev)并未通過Ranger Admin在HDFS插件里的服務管理里注冊成服務(名hadoopdev).
解決方案:
1 登錄Ranger Adming
2 點擊HDFS plugin的添加按鈕
3 定義服務名為hadoopdev,提交其它信息后保存
#再次驗證HDFS plugin插件,則發現已經正常加載
注:
1 如果沒有安裝和開啟Ranger-usersync服務的情況下直接測試HDFS賦權權限是不成功的.
2 Ranger Admin的日志文件見 /root/ranger-0.5.0-admin/ews/logs/xa_portal.log
4.5 Hive-Plugin安裝配置
先啟動hive的metastore和hiveserver2服務
nohup hive --service metastore-hiveconf hive.root.logger=INFO,console > myout1.file 2>&1 &
nohup hiveserver2 -hiveconfhive.root.logger=INFO,console > myout2.file 2>&1 &
#beeline驗證
[root@localhost ~]# beeline -u"jdbc:hive2://192.168.56.101:10000" -n root -p test
Connectingto jdbc:hive2://192.168.56.101:10000
Connectedto: Apache Hive (version 1.2.1)
Driver:Hive JDBC (version 1.2.1)
Transactionisolation: TRANSACTION_REPEATABLE_READ
Beelineversion 1.2.1 by Apache Hive
0:jdbc:hive2://192.168.56.101:10000> show databases;
+----------------+--+
|database_name? |
+----------------+--+
|default??????? |
| shenl????????? |
+----------------+--+
?
2) Ranger-Admin里注冊hive plugin的服務
?
3) 配置、啟用hive plugin
cp/root/incubator-ranger/target/ranger-0.5.0-hive-plugin.tar.gz /root
cd /root
tar -zxvf ranger-0.5.0-hive-plugin.tar.gz
cd ranger-0.5.0-hive-plugin
#打開Hive Plugin里install.properties文件,修改參數的值如下所示:
vi install.properties
POLICY_MGR_URL=http://192.168.56.101:6080
REPOSITORY_NAME=hivedev
XAAUDIT.DB.IS_ENABLED=true
XAAUDIT.DB.FLAVOUR=MYSQL=MYSQL
XAAUDIT.DB.HOSTNAME=localhost
XAAUDIT.DB.DATABASE_NAME=ranger_audit
XAAUDIT.DB.USER_NAME=root
XAAUDIT.DB.PASSWORD=root
?
[root@localhostranger-0.5.0-hive-plugin]# ./enable-hive-plugin.sh
Customuser and group is available, using custom user and group.
ERROR:Unable to find the conf directory of component [hive]; dir [/root/hive/conf]not found.
Exitinginstallation.
#解決方法:
ln -s /root/apache-hive-0.13.0-bin/conf//root/hive/conf
?
[root@localhostranger-0.5.0-hive-plugin]# ./enable-hive-plugin.sh
Customuser and group is available, using custom user and group.
ERROR:Unable to find the lib directory of component [hive];? dir [/root/hive/lib] not found.
Exitinginstallation.
#解決方法:
cp root/ranger-0.5.0-hive-plugin/lib/ranger-hive-plugin-impl/*.jar/root/apache-hive-1.2.1-bin/lib/
mkdir /root/hive/lib
ln -s/root/apache-hive-1.2.1-bin/lib/ /root/hive/lib/
#將生成的hiveserver2-site.xml拷貝到hive的配置目錄下
cp /root/hive/conf/*/root/apache-hive-1.2.1-bin/conf/
#如果hive配置目錄里有hiveserver2-site.xml,則需要添加如下內容:
<property>
???????<name>hive.security.authorization.enabled</name>
??????? <value>true</value>
??? </property>
??? <property>
???????<name>hive.security.authorization.manager</name>
??????? <value>org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory</value>
??? </property>
??? <property>
???????<name>hive.security.authenticator.manager</name>
???????<value>org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator</value>
??? </property>
??? <property>
???????<name>hive.conf.restricted.list</name>
???????<value>hive.security.authorization.enabled,hive.security.authorization.manager,hive.security.authenticator.manager</value>
??? </property>
5 使用
5.1 HDFS Plugin驗證
1 新增linux用戶shenl進行測試
useradd shenl
hadoop fs –mkdir /shenl
2 登錄Ranger Admin,新增用戶shenl
3 登錄Ranger Admin,在hadoopdev里注冊新的Policy
4 添加該Policy的權限,指定可以對HDFS里的/shenl有R權限
#切換到shenl用戶進行put權限測試,應該報錯.
5 編輯Policy追加shenl用戶的W權限
#此時shenl用戶應該可以擁有/shenl目錄的put權限
5.2 Hive Plugin驗證
1 hive plugin加載驗證
2 定義權限策略
3 beeline里權限驗證
6 總結
無 與50位技術專家面對面20年技術見證，附贈技術全景圖總結
以上是生活随笔為你收集整理的Apache Ranger源码编译及使用的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 宏? bios怎么设置 宏? BIOS设
- 下一篇： CDH Hadoop 基于CM方式半在线