1：上傳文件到hdfs

[jifeng@feng01 hadoop]$ hadoop fs -mkdir /user/jifeng/fire 15/03/05 13:29:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [jifeng@feng01 hadoop]$ hadoop fs -put /home/jifeng/hadoop/firewallb-week2-origin.sql /user/jifeng/fire/ 15/03/05 13:30:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [jifeng@feng01 hadoop]$
2:hive中建表

[jifeng@feng01 ~]$ hive 15/03/05 13:31:08 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect. Use hive.hmshandler.retry.* insteadLogging initialized using configuration in jar:file:/home/jifeng/hive/lib/hive-common-0.13.1.jar!/hive-log4j.properties hive> CREATE EXTERNAL TABLE firewall ( idauto INT, dt TIMESTAMP, dtstr STRING, timeSeconds INT, priority STRING, operation STRING, messageCode STRING, protocol STRING, srcIp STRING, destIp STRING, srcIPNumber bigINT, destIPNumber bigINT, srcPort INT, destPort INT, destService STRING, direction STRING, flags STRING, command STRING, slot1min INT, slot5min INT, slot30min INT, slot60min INT ) row format delimited fields terminated by '\t'; OK Time taken: 3.414 seconds hive> load data inpath '/user/jifeng/fire/firewallb-week2-origin.sql' into table firewall; Loading data to table default.firewall Table default.firewall stats: [numFiles=1, numRows=0, totalSize=3415284428, rawDataSize=0] OK Time taken: 1.441 seconds

建表語(yǔ)句：

CREATE EXTERNAL TABLE firewall ( idauto INT, dt TIMESTAMP, dtstr STRING, timeSeconds INT, priority STRING, operation STRING, messageCode STRING, protocol STRING, srcIp STRING, destIp STRING, srcIPNumber bigINT, destIPNumber bigINT, srcPort INT, destPort INT, destService STRING, direction STRING, flags STRING, command STRING, slot1min INT, slot5min INT, slot30min INT, slot60min INT ) row format delimited fields terminated by '\t';

3：hive中測(cè)試

hive> select * from firewall order by dt desc limit 10; Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes):set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers:set hive.exec.reducers.max=<number> In order to set a constant number of reducers:set mapreduce.job.reduces=<number> Starting Job = job_1423061535604_0004, Tracking URL = http://feng01:8088/proxy/application_1423061535604_0004/ Kill Command = /home/jifeng/hadoop/hadoop-2.4.1/bin/hadoop job -kill job_1423061535604_0004 Hadoop job information for Stage-1: number of mappers: 13; number of reducers: 1 2015-03-05 13:35:57,706 Stage-1 map = 0%, reduce = 0% 2015-03-05 13:36:58,247 Stage-1 map = 0%, reduce = 0% 2015-03-05 13:37:26,248 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 5.38 sec 2015-03-05 13:37:34,542 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 7.63 sec 2015-03-05 13:38:40,840 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 13.89 sec 2015-03-05 13:39:24,926 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 79.08 sec 2015-03-05 13:40:00,669 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 85.37 sec 2015-03-05 13:40:25,540 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 86.27 sec 2015-03-05 13:40:45,643 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 107.73 sec 2015-03-05 13:40:46,742 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 109.79 sec 2015-03-05 13:40:59,426 Stage-1 map = 36%, reduce = 0%, Cumulative CPU 116.75 sec 2015-03-05 13:41:03,706 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 120.27 sec 2015-03-05 13:41:07,921 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 125.38 sec 2015-03-05 13:41:15,830 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 130.26 sec 2015-03-05 13:41:18,002 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 131.86 sec 2015-03-05 13:42:18,780 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 170.98 sec 2015-03-05 13:43:19,428 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 194.07 sec 2015-03-05 13:44:20,028 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 197.32 sec 2015-03-05 13:45:20,494 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 200.93 sec 2015-03-05 13:46:21,209 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 223.6 sec 2015-03-05 13:46:42,075 Stage-1 map = 54%, reduce = 15%, Cumulative CPU 230.03 sec 2015-03-05 13:46:49,284 Stage-1 map = 56%, reduce = 15%, Cumulative CPU 218.59 sec 2015-03-05 13:46:51,358 Stage-1 map = 60%, reduce = 15%, Cumulative CPU 219.07 sec 2015-03-05 13:46:54,448 Stage-1 map = 63%, reduce = 15%, Cumulative CPU 191.15 sec 2015-03-05 13:46:58,609 Stage-1 map = 64%, reduce = 15%, Cumulative CPU 193.67 sec 2015-03-05 13:47:05,823 Stage-1 map = 67%, reduce = 15%, Cumulative CPU 180.53 sec 2015-03-05 13:47:09,938 Stage-1 map = 69%, reduce = 15%, Cumulative CPU 194.45 sec 2015-03-05 13:47:11,986 Stage-1 map = 69%, reduce = 18%, Cumulative CPU 194.85 sec 2015-03-05 13:47:47,100 Stage-1 map = 72%, reduce = 18%, Cumulative CPU 209.86 sec 2015-03-05 13:48:31,364 Stage-1 map = 77%, reduce = 18%, Cumulative CPU 210.55 sec 2015-03-05 13:48:37,549 Stage-1 map = 77%, reduce = 21%, Cumulative CPU 210.74 sec 2015-03-05 13:48:53,998 Stage-1 map = 82%, reduce = 21%, Cumulative CPU 177.87 sec 2015-03-05 13:48:55,021 Stage-1 map = 82%, reduce = 23%, Cumulative CPU 178.51 sec 2015-03-05 13:49:06,447 Stage-1 map = 90%, reduce = 23%, Cumulative CPU 143.53 sec 2015-03-05 13:49:11,591 Stage-1 map = 90%, reduce = 26%, Cumulative CPU 144.64 sec 2015-03-05 13:49:12,612 Stage-1 map = 95%, reduce = 26%, Cumulative CPU 145.78 sec 2015-03-05 13:49:14,665 Stage-1 map = 95%, reduce = 28%, Cumulative CPU 145.81 sec 2015-03-05 13:49:28,043 Stage-1 map = 97%, reduce = 28%, Cumulative CPU 147.61 sec 2015-03-05 13:49:29,075 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 147.67 sec 2015-03-05 13:50:06,153 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 148.22 sec 2015-03-05 13:50:09,249 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 149.17 sec MapReduce Total cumulative CPU time: 2 minutes 29 seconds 170 msec Ended Job = job_1423061535604_0004 MapReduce Jobs Launched: Job 0: Map: 13 Reduce: 1 Cumulative CPU: 149.68 sec HDFS Read: 3418565375 HDFS Write: 2076 SUCCESS Total MapReduce CPU Time Spent: 2 minutes 29 seconds 680 msec OK 16600916 2013-04-15 10:00:00 15/Apr/2013 10:00:00 1366020000 Warning Deny ASA-4-106023 TCP 10.6.6.7 10.0.2.2 168166919 167772674 42465 52372 52372_tcp (empty) (empty) (empty) 227670004553400 758900 379450 16600915 2013-04-15 10:00:00 15/Apr/2013 10:00:00 1366020000 Warning Deny ASA-4-106023 TCP 10.13.77.49 10.0.3.2 168643889 167772930 58200 32179 32179_tcp (empty) (empty) (empty) 22767000 4553400 758900 379450 16600931 2013-04-15 10:00:00 15/Apr/2013 10:00:00 1366020000 Warning Deny ASA-4-106023 TCP 10.6.6.7 10.0.2.2 168166919 167772674 42464 33004 33004_tcp (empty) (empty) (empty) 227670004553400 758900 379450 16600919 2013-04-15 10:00:00 15/Apr/2013 10:00:00 1366020000 Warning Deny ASA-4-106023 TCP 10.6.6.7 10.0.2.2 168166919 167772674 42465 38300 38300_tcp (empty) (empty) (empty) 227670004553400 758900 379450 16600918 2013-04-15 10:00:00 15/Apr/2013 10:00:00 1366020000 Warning Deny ASA-4-106023 TCP 10.13.77.49 10.0.3.2 168643889 167772930 58199 25087 25087_tcp (empty) (empty) (empty) 22767000 4553400 758900 379450 16600912 2013-04-15 10:00:00 15/Apr/2013 10:00:00 1366020000 Warning Deny ASA-4-106023 TCP 10.6.6.7 10.0.2.2 168166919 167772674 42465 556 remotefs (empty) (empty) (empty) 227670004553400 758900 379450 16600913 2013-04-15 10:00:00 15/Apr/2013 10:00:00 1366020000 Warning Deny ASA-4-106023 TCP 10.13.77.49 10.0.3.2 168643889 167772930 58199 32179 32179_tcp (empty) (empty) (empty) 22767000 4553400 758900 379450 16600920 2013-04-15 10:00:00 15/Apr/2013 10:00:00 1366020000 Info Teardown ASA-6-302014 TCP 10.6.6.7 172.10.0.2 168166919 2886336514 42706 3389 3389_tcp inbound SYN Timeout (empty) 22767000 4553400 758900 379450 16600930 2013-04-15 10:00:00 15/Apr/2013 10:00:00 1366020000 Warning Deny ASA-4-106023 TCP 10.13.77.49 10.0.3.2 168643889 167772930 58200 13817 13817_tcp (empty) (empty) (empty) 22767000 4553400 758900 379450 16600911 2013-04-15 10:00:00 15/Apr/2013 10:00:00 1366020000 Warning Deny ASA-4-106023 TCP 10.13.77.49 10.0.3.2 168643889 167772930 58200 1993 1993_tcp (empty) (empty) (empty) 22767000 4553400 758900 379450 Time taken: 879.31 seconds, Fetched: 10 row(s)

4：啟動(dòng)spark sql server

[jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./sbin/start-thriftserver.sh --hiveconf hive.server2.thrift.port=10000 --hiveconf hive.server2.thrift.bind.host=feng02 --master spark://feng02:7077 starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2, logging to /home/jifeng/hadoop/spark-1.2.0-bin-2.4.1/sbin/../logs/spark-jifeng-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-feng02.out
運(yùn)行beeline

[jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./bin/beeline Spark assembly has been built with Hive, including Datanucleus jars on classpath Beeline version ??? by Apache Hive
連接：

beeline> !connect jdbc:hive2://feng02:10000 scott tiger org.apache.hive.jdbc.HiveDriver Connecting to jdbc:hive2://feng02:10000 log4j:WARN No appenders could be found for logger (org.apache.thrift.transport.TSaslTransport). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Connected to: Spark SQL (version 1.2.0) Driver: null (version null) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://feng02:10000> show tables; +----------------+ | result | +----------------+ | course | | firewall | | hbase_table_1 | | pokes | | student | +----------------+ 5 rows selected (0.489 seconds)

5：spark sql測(cè)試

0: jdbc:hive2://feng02:10000> select * from firewall order by dt desc limit 10; +-----------+------------------------+-----------------------+--------------+-----------+------------+---------------+--+ | idauto | dt | dtstr | timeseconds | priority | operation | messagecode | | +-----------+------------------------+-----------------------+--------------+-----------+------------+---------------+--+ | 16600918 | 2013-04-15 10:00:00.0 | 15/Apr/2013 10:00:00 | 1366020000 | Warning | Deny | ASA-4-106023 | | | 16600914 | 2013-04-15 10:00:00.0 | 15/Apr/2013 10:00:00 | 1366020000 | Warning | Deny | ASA-4-106023 | | | 16600917 | 2013-04-15 10:00:00.0 | 15/Apr/2013 10:00:00 | 1366020000 | Warning | Deny | ASA-4-106023 | | | 16600912 | 2013-04-15 10:00:00.0 | 15/Apr/2013 10:00:00 | 1366020000 | Warning | Deny | ASA-4-106023 | | | 16600913 | 2013-04-15 10:00:00.0 | 15/Apr/2013 10:00:00 | 1366020000 | Warning | Deny | ASA-4-106023 | | | 16600915 | 2013-04-15 10:00:00.0 | 15/Apr/2013 10:00:00 | 1366020000 | Warning | Deny | ASA-4-106023 | | | 16600916 | 2013-04-15 10:00:00.0 | 15/Apr/2013 10:00:00 | 1366020000 | Warning | Deny | ASA-4-106023 | | | 16600911 | 2013-04-15 10:00:00.0 | 15/Apr/2013 10:00:00 | 1366020000 | Warning | Deny | ASA-4-106023 | | | 16600920 | 2013-04-15 10:00:00.0 | 15/Apr/2013 10:00:00 | 1366020000 | Info | Teardown | ASA-6-302014 | | | 16600919 | 2013-04-15 10:00:00.0 | 15/Apr/2013 10:00:00 | 1366020000 | Warning | Deny | ASA-4-106023 | | +-----------+------------------------+-----------------------+--------------+-----------+------------+---------------+--+ 10 rows selected (119.692 seconds)

統(tǒng)計(jì)IP出現(xiàn)次數(shù)：

0: jdbc:hive2://feng02:10000> select srcIp,count(*) as num from firewall group by srcIp order by num desc limit 100; +-----------------+----------+ | srcIp | num | +-----------------+----------+ | 10.12.15.152 | 3875019 | | 10.13.77.49 | 1954742 | | 10.6.6.7 | 1769857 | | 10.138.235.111 | 1630920 | | 10.12.14.15 | 756475 | | 10.17.15.10 | 720936 | | 172.10.0.4 | 482568 | | 10.78.100.150 | 300530 | | 10.247.58.182 | 292384 | | 10.38.217.48 | 282588 | | 172.30.0.4 | 232850 | | 10.10.11.102 | 205579 | | 10.170.32.110 | 202735 | | 10.170.32.181 | 200539 | | 10.138.214.18 | 197864 | | 10.247.106.27 | 193912 | | 10.156.165.120 | 183929 | | 10.15.7.85 | 172422 | | 10.0.0.42 | 165063 | | 10.200.20.2 | 164519 | | 10.70.68.127 | 161735 | | 172.20.0.15 | 145924 | | 10.250.178.101 | 140053 | | 172.20.0.4 | 122768 | | 172.30.1.223 | 51329 | | 172.20.1.81 | 51328 | | 172.10.2.66 | 51304 | | 172.20.1.47 | 51298 | | 172.30.1.218 | 51236 | | 172.10.2.135 | 51233 | | 172.20.1.23 | 51199 | | 172.10.2.106 | 51174 | | 172.20.0.3 | 20795 | | 172.10.0.2 | 16296 | | 10.13.77.4 | 15622 | | 10.77.101.2 | 15307 | | 10.77.24.4 | 15274 | | 10.0.2.181 | 15264 | | 10.0.3.77 | 15086 | | 10.58.99.97 | 15021 | | 10.83.101.25 | 14997 | | 10.62.15.101 | 14930 | | 10.1.55.43 | 14783 | | 172.20.0.2 | 14754 | | 10.27.200.49 | 14690 | | 10.18.37.65 | 14645 | | 10.43.51.79 | 14457 | | 172.30.0.3 | 11113 | | 172.30.0.2 | 10129 | | 172.10.0.3 | 9598 | | 172.10.0.9 | 3899 | | 172.10.0.8 | 3852 | | 172.20.0.6 | 3833 | | 172.10.0.7 | 3820 | | 172.10.0.5 | 3805 | | 10.57.84.113 | 3420 | | 10.57.105.222 | 3329 | | 10.170.32.239 | 3231 | | 172.30.1.215 | 2973 | | 172.30.1.248 | 2908 | | 172.30.1.213 | 2890 | | 172.30.1.217 | 2713 | | 172.30.1.210 | 2686 | | 172.20.0.5 | 2668 | | 172.30.1.205 | 2639 | | 10.76.187.67 | 2609 | | 172.30.1.204 | 2556 | | 172.30.1.227 | 2553 | | 172.30.1.242 | 2525 | | 172.30.1.236 | 2521 | | 10.156.165.212 | 2513 | | 10.88.52.24 | 2513 | | 172.30.1.247 | 2513 | | 10.41.9.89 | 2509 | | 10.89.101.53 | 2503 | | 10.88.25.4 | 2494 | | 172.30.1.243 | 2473 | | 172.30.1.245 | 2457 | | 172.30.1.241 | 2442 | | 10.51.88.167 | 2418 | | 172.30.1.202 | 2415 | | 172.30.1.237 | 2414 | | 10.0.3.178 | 2414 | | 10.36.156.99 | 2411 | | 10.32.42.116 | 2410 | | 10.78.90.9 | 2408 | | 172.30.1.229 | 2408 | | 10.33.0.15 | 2406 | | 10.0.125.6 | 2400 | | 10.138.78.75 | 2400 | | 10.51.88.253 | 2400 | | 10.38.37.13 | 2398 | | 10.76.187.68 | 2397 | | 10.76.193.186 | 2397 | | 10.149.59.54 | 2396 | | 10.170.148.52 | 2395 | | 10.179.19.15 | 2395 | | 10.57.105.44 | 2395 | | 172.30.1.224 | 2394 | +-----------------+----------+ | srcIp | num | +-----------------+----------+ | 10.38.37.229 | 2393 | +-----------------+----------+ 100 rows selected (65.642 seconds)

0: jdbc:hive2://feng02:10000> select count(*) from firewall; +-----------+ | _c0 | +-----------+ | 16600931 | +-----------+ 1 row selected (47.636 seconds)

總結(jié)

以上是生活随笔為你收集整理的spark sql and hive 3g数据测试的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。