Linux執行 Hadoop WordCount

Ubuntu 終端進入快捷鍵 ：ctrl + Alt +t

hadoop啟動命令：start-all.sh

正常執行效果如下：

hadoop@HADOOP:~$ start-all.sh

Warning: $HADOOP_HOME is deprecated.

starting namenode, logging to /home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-namenode-HADOOP.MAIN.out

HADOOP.MAIN: starting datanode, logging to/home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-datanode-HADOOP.MAIN.out

HADOOP.MAIN: starting secondarynamenode,logging to/home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-secondarynamenode-HADOOP.MAIN.out

starting jobtracker, logging to/home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-jobtracker-HADOOP.MAIN.out

HADOOP.MAIN: starting tasktracker, loggingto /home/hadoop/hadoop-1.1.2/libexec/../logs/hadoop-hadoop-tasktracker-HADOOP.MAIN.out

jps命令查看啟動的hadoop服務

hadoop@HADOOP:~$jps

3615 Jps

2699 NameNode

3461 TaskTracker

2922 DataNode

3137 SecondaryNameNode

3231 JobTracker

本地創建一個文件夾

hadoop@HADOOP:~$ mkdir ~/file

在file文件創建兩個txt文件

hadoop@HADOOP:~$ cd file

hadoop@HADOOP:~/file$ echo "Hello World" > file1.txt

hadoop@HADOOP:~/file$ echo "Hello Hadoop" > file2.txt

hadoop@HADOOP:~/file$ ls

file1.txt?file2.txt

hadoop@HADOOP:~/file$

在HDFS上創建一個輸入文件夾

hadoop@HADOOP:~/file$ hadoop fs -mkdir input

查看創建的input文件夾路徑

hadoop@HADOOP:~$ hadoop fs -ls

Warning: $HADOOP_HOME is deprecated.

Found 5 items

-rw-r--r--? 3 Administrator supergroup? ?6296230 2014-09-03 10:38 /user/hadoop/cloud.txt

drwxr-xr-x? - hadoop? ? ? ? supergroup? ? ? ? ? 0 2014-09-02 16:31/user/hadoop/hadi_curbm

drwxr-xr-x? - hadoop? ? ? ? supergroup? ? ? ? ? 0 2014-09-04 09:59/user/hadoop/input

drwxr-xr-x? - hadoop? ? ? ? supergroup? ? ? ? ? 0 2014-09-02 16:31/user/hadoop/pegasus

hadoop@HADOOP:~$

可以看到目錄被創建到 /user/hadoop/input 目錄

上傳本地file文件到input目錄

hadoop@HADOOP:~$hadoop fs put ~/file/*.txt /user/hadoop/input

找到hadoop目錄下的examples.jar 程序包

hadoop@HADOOP:~$ cd hadoop-1.1.2

hadoop@HADOOP:~/hadoop-1.1.2$ ls

bin? ? ? ? ?docs? ? ? ? ? ? ? ? ? ? ? ? ?hadoop-test-1.1.2.jar? LICENSE.txt?src

build.xml? ?hadoop-ant-1.1.2.jar? ? ? ? ?hadoop-tools-1.1.2.jar? logs? ? ? ? webapps

c++? ? ? ? ?hadoop-client-1.1.2.jar? ? ? ivy? ? ? ? ? ? ? ? ? ? NOTICE.txt? wordcount.jar

CHANGES.txt?hadoop-core-1.1.2.jar? ? ? ? ivy.xml? ? ? ? ? ? ? ? README.txt

conf? ? ? ? hadoop-examples-1.1.2.jar? ? lib? ? ? ? ? ? ? ? ? ? sbin

contrib? ? ?hadoop-minicluster-1.1.2.jar?libexec? ? ? ? ? ? ? ? share

hadoop@HADOOP:~/hadoop-1.1.2$

執行jar程序代碼 統計input目錄下文件的Wordcount

hadoop@HADOOP:~$ hadoop jar /home/hadoop/hadoop-1.1.2/hadoop-examples-1.1.2.jar wordcount ?/user/hadoop/input output

Warning: $HADOOP_HOME is deprecated.

14/09/04 10:10:44 INFOinput.FileInputFormat: Total input paths to process : 0

14/09/04 10:10:45 INFO mapred.JobClient:Running job: job_201409040943_0001

14/09/04 10:10:46 INFOmapred.JobClient:? map 0% reduce 0%

14/09/04 10:10:54 INFOmapred.JobClient:? map 0% reduce 100%

14/09/04 10:10:55 INFO mapred.JobClient:Job complete: job_201409040943_0001

14/09/04 10:10:55 INFO mapred.JobClient:Counters: 18

14/09/04 10:10:55 INFOmapred.JobClient:? Job Counters

14/09/04 10:10:55 INFOmapred.JobClient:? ? Launched reducetasks=1

14/09/04 10:10:55 INFOmapred.JobClient:? ? SLOTS_MILLIS_MAPS=4087

14/09/04 10:10:55 INFOmapred.JobClient:? ? Total time spent byall reduces waiting after reserving slots (ms)=0

14/09/04 10:10:55 INFOmapred.JobClient:? ? Total time spent byall maps waiting after reserving slots (ms)=0

14/09/04 10:10:55 INFO mapred.JobClient:? ? SLOTS_MILLIS_REDUCES=4068

14/09/04 10:10:55 INFOmapred.JobClient:? File Output FormatCounters

14/09/04 10:10:55 INFOmapred.JobClient:? ? Bytes Written=0

14/09/04 10:10:55 INFOmapred.JobClient:? FileSystemCounters

14/09/04 10:10:55 INFO mapred.JobClient:? ? FILE_BYTES_WRITTEN=55309

14/09/04 10:10:55 INFOmapred.JobClient:? Map-Reduce Framework

14/09/04 10:10:55 INFOmapred.JobClient:? ? Reduce inputgroups=0

14/09/04 10:10:55 INFOmapred.JobClient:? ? Combine outputrecords=0

14/09/04 10:10:55 INFOmapred.JobClient:? ? Reduce shufflebytes=0

14/09/04 10:10:55 INFOmapred.JobClient:? ? Physical memory(bytes) snapshot=35037184

14/09/04 10:10:55 INFOmapred.JobClient:? ? Reduce outputrecords=0

14/09/04 10:10:55 INFO mapred.JobClient: ? ? Spilled Records=0

14/09/04 10:10:55 INFOmapred.JobClient:? ? CPU time spent(ms)=120

14/09/04 10:10:55 INFOmapred.JobClient:? ? Total committedheap usage (bytes)=15925248

14/09/04 10:10:55 INFOmapred.JobClient:? ? Virtual memory(bytes) snapshot=377499648

14/09/04 10:10:55 INFOmapred.JobClient:? ? Combine inputrecords=0

14/09/04 10:10:55 INFOmapred.JobClient:? ? Reduce inputrecords=0

hadoop@HADOOP:~$

顯示結果

hadoop@HADOOP:~$ hadoop fs -ls output

Warning: $HADOOP_HOME is deprecated.

Found 3 items

-rw-r--r--? 1 hadoop supergroup? ? ? ? ? 02014-09-04 10:10 /user/hadoop/output/_SUCCESS

drwxr-xr-x? - hadoop supergroup? ? ? ? ? 02014-09-04 10:10 /user/hadoop/output/_logs

-rw-r--r--? 1 hadoop supergroup? ? ? ? ? 02014-09-04 10:10/user/hadoop/output/part-r-00000

hadoop@HADOOP:~$

查看執行結果

hadoop@HADOOP:~$ hadoop fs -cat output/part-r-00000

Hadoop?1

Hello? ?2

World? 1

總結

以上是生活随笔為你收集整理的linux hadoop 运行jar,Linux下执行Hadoop WordCount.jar的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。