hadoop-0.21.0 在Windows环境下的部署(2)Hadoop配置
生活随笔
收集整理的這篇文章主要介紹了
hadoop-0.21.0 在Windows环境下的部署(2)Hadoop配置
小編覺得挺不錯的,現(xiàn)在分享給大家,幫大家做個參考.
1：下載
本來想傳到csdn上的，但是最大只支持70M的文件，還是網(wǎng)上找吧
2：解壓在C:\cygwin\hadoop
3：配置
hadoop/conf下需要配置的文件：
1.hadoop-env.sh
修改JDK的路徑 export JAVA_HOME
# Set Hadoop-specific environment variables here.# The only required environment variable is JAVA_HOME. All others are # optional. When running a distributed configuration it is best to # set JAVA_HOME in this file, so that it is correctly defined on # remote nodes.# The java implementation to use. Required. export JAVA_HOME=/cygdrive/d/java/Tomcat6/jdk# Extra Java CLASSPATH elements. Optional. # export HADOOP_CLASSPATH=2.core-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Put site-specific property overrides in this file. --><configuration> <property><name>fs.default.name</name><value>hdfs://localhost:9000</value> </property> </configuration>
3.hdfs-site.xml
<value>1</value> 該變量意思是文件系統(tǒng)中文件的復(fù)本數(shù)量。在單獨(dú)的一個數(shù)據(jù)節(jié)點(diǎn)上運(yùn)行時，HDFS無法將塊復(fù)制到三個數(shù)據(jù)節(jié)點(diǎn)上。
4.mapred-site.xml
4：啟動Hadoop
第一步，在hadoop目錄下創(chuàng)建目錄logs，用于保存日志
第二步，Format a new distributed-filesystem: 格式化namenode，創(chuàng)建HDFS
執(zhí)行命令： bin/hadoop namenode -format
如果有錯誤請參考：org.apache.hadoop.util.PlatformName //cgywin下Hadoop-0.21.0 錯誤問題
lenovo@lenovo-PC /hadoop $ bin/hadoop namenode -format DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it.[INFO ][mgmnt ] Local JMX connector started 13/07/02 09:43:50 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = lenovo-PC/10.6.3.180 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 0.21.0 STARTUP_MSG: classpath = C:\cygwin\hadoop\conf;D:\java\Tomcat6\jdk\lib\tools.jar;C:\cygwin\hadoop\;C:\cygwin\hadoop\hadoop-common-0.21.0.jar;C:\cygwin\hadoop\...... STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -r 985326; compiled by 'tomwhite' on Tue Aug 17 01:02:28 EDT 2010 ************************************************************/ 13/07/02 09:43:51 INFO namenode.FSNamesystem: defaultReplication = 1 13/07/02 09:43:51 INFO namenode.FSNamesystem: maxReplication = 512 13/07/02 09:43:51 INFO namenode.FSNamesystem: minReplication = 1 13/07/02 09:43:51 INFO namenode.FSNamesystem: maxReplicationStreams = 2 13/07/02 09:43:51 INFO namenode.FSNamesystem: shouldCheckForEnoughRacks = false 13/07/02 09:43:51 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000 13/07/02 09:43:51 INFO namenode.FSNamesystem: fsOwner=lenovo 13/07/02 09:43:51 INFO namenode.FSNamesystem: supergroup=supergroup 13/07/02 09:43:51 INFO namenode.FSNamesystem: isPermissionEnabled=true 13/07/02 09:43:51 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 13/07/02 09:43:51 INFO common.Storage: Image file of size 112 saved in 0 seconds . 13/07/02 09:43:52 INFO common.Storage: Storage directory \tmp\hadoop-lenovo\dfs\name has been successfully formatted. 13/07/02 09:43:52 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at lenovo-PC/10.6.3.180 ************************************************************/第三步，啟動Hadoop，執(zhí)行命令： bin/start-all.sh
lenovo@lenovo-PC /hadoop $ bin/start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-mapred.sh starting namenode, logging to C:\cygwin\hadoop\logs/hadoop-lenovo-namenode-lenovo-PC.out lenovo@localhost's password: localhost: starting datanode, logging to C:\cygwin\hadoop\logs/hadoop-lenovo-datanode-lenovo-PC.out lenovo@localhost's password: localhost: starting secondarynamenode, logging to C:\cygwin\hadoop\logs/hadoop-lenovo-secondarynamenode-lenovo- PC.out starting jobtracker, logging to C:\cygwin\hadoop\logs/hadoop-lenovo-jobtracker-lenovo-PC.out lenovo@localhost's password: localhost: starting tasktracker, logging to C:\cygwin\hadoop\logs/hadoop-lenovo-tasktracker-lenovo-PC.out還需要輸入3次密碼，運(yùn)行5個JVM進(jìn)程，查看進(jìn)程：
?
lenovo@lenovo-PC /hadoop $ psPID PPID PGID WINPID TTY UID STIME COMMAND29756 1 30500 30576 ? 1000 10:02:19 /cygdrive/d/java/Tomcat6/jdk/bin/java28184 1 21636 26008 pty0 1000 10:01:52 /cygdrive/d/java/Tomcat6/jdk/bin/java23856 1 21636 22664 pty0 1000 10:01:11 /cygdrive/d/java/Tomcat6/jdk/bin/java31320 4868 31320 31332 pty0 1000 10:03:38 /usr/bin/ps27224 1 19108 27980 ? 1000 10:01:49 /cygdrive/d/java/Tomcat6/jdk/bin/java25556 1 24812 26308 ? 1000 10:01:27 /cygdrive/d/java/Tomcat6/jdk/bin/java8732 1 8732 8732 ? 1000 09:12:27 /usr/bin/mintty4868 8732 4868 9376 pty0 1000 09:12:27 /usr/bin/bash運(yùn)行成功。
?
命令：
1)：bin/hadoop fs -copyFromLocal local/hibernate.rar /user/pdf/hibernate.rar
上傳本地文件到HDFS
lenovo@lenovo-PC /hadoop $ bin/hadoop fs -copyFromLocal local/hibernate.rar /user/pdf/hibernate.rar 13/07/02 10:15:06 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000 13/07/02 10:15:06 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id2)：查詢上傳的文件
lenovo@lenovo-PC /hadoop $ bin/hadoop fs -ls /user/pdf 13/07/02 15:57:26 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000 13/07/02 15:57:26 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id Found 1 items -rw-r--r-- 1 lenovo supergroup 3375455 2013-07-02 10:15 /user/pdf/hibernate.rar也可以在web中查詢：
?hadoop默認(rèn)的NameNode 和 JobTracker
- NameNode?-?http://localhost:50070/
- JobTracker?-?http://localhost:50030/
總結(jié)
以上是生活随笔為你收集整理的hadoop-0.21.0 在Windows环境下的部署(2)Hadoop配置的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： hadoop-0.21.0 在Windo
- 下一篇： java信息管理系统总结_java实现科