0、preliminary 環(huán)境搭建

Setup development environment

Download the latest version of MRUnit jar from Apache website: https://repository.apache.org/content/repositories/releases/org/apache/mrunit/mrunit/. For example if you are using the Hadoop version 1.0.3, download mrunit-x.x.x-incubating-hadoop2.jar.
Include the jar in your IDE classpath. Also download the latest verison of mokito (http://code.google.com/p/mockito/) and JUnit jar and add them to your class path of development environment.
請導入jar包到CentOS中的eclipse

download site: https://mrunit.apache.org/general/downloads.html

download web site https://repository.apache.org/content/repositories/releases/org/apache/mrunit/mrunit/1.1.0/

You should Attention:

hadoop 框架包里面有自帶的mock jar包，吧他們?nèi)縿h除，否則要產(chǎn)生jar包兼容性異常（編譯器不曉得調(diào)哪個jar包為好）

---

轉(zhuǎn)自： http://www.infoq.com/cn/articles/HadoopMRUnit

(請關心里面的干貨——測試用例文末)

引言

Hadoop MapReduce作業(yè)有著獨一無二的代碼架構，這種代碼架構擁有特定的模板和結(jié)構。這樣的架構會給測試驅(qū)動開發(fā)和單元測試帶來一些麻煩。這篇文章是運用MRUnit，Mockito和PowerMock的真實范例。

我會介紹

使用MRUnit來編寫Hadoop MapReduce應用程序的JUnit測試
使用PowerMock和Mockito模擬靜態(tài)方法
模擬其他類型中的業(yè)務邏輯（譯注：也就是編寫測試驅(qū)動模塊）
查看模擬的業(yè)務邏輯是否被調(diào)用（譯注：測試驅(qū)動模塊是否運行正常）
計數(shù)器
測試用例與log4j的集成
異常處理

本文的前提是讀者應該已經(jīng)熟悉JUnit 4的使用。

使用MRUnit可以把測試樁輸入到mapper和/或reducer中，然后在JUnit環(huán)境中判斷是否通過測試。這個過程和任何JUnit測試一樣，你可以調(diào)試你的代碼。MRUnit中的MapReduce Driver可以測試一組Map/Reduce或者Combiner。 PipelineMapReduceDriver可以測試Map/Reduce作業(yè)工作流。目前，MRUnit還沒有Partitioner對應的驅(qū)動。MRUnit使開發(fā)人員在面對Hadoop特殊的架構的時候也能進行TDD和輕量級的單元測試。

測試用例（MapTest + ReduceTest + MapReduceTest）

publicclass MapTest {

private Mapper mapper; private MapDriver driver;@Before publicvoid init(){mapper = new WordCount.Map();driver = new MapDriver(mapper); }@Test publicvoid test() throws IOException{String line = "this is a test case for map";driver.withInput(new LongWritable(1),new Text(line)).withOutput(new Text("this"), new IntWritable(1)).withOutput(new Text("is"), new IntWritable(1)).withOutput(new Text("a"), new IntWritable(1)).withOutput(new Text("test"), new IntWritable(1)).withOutput(new Text("case"), new IntWritable(1)).withOutput(new Text("for"), new IntWritable(1)).withOutput(new Text("map"), new IntWritable(1)).runTest(); }

}

publicclass ReduceTest {

private Reducer reducer; private ReduceDriver driver;@Before publicvoid init(){reducer = new WordCount.Reduce();driver = new ReduceDriver(reducer);} @Test publicvoid test() throws IOException{String key = "test";List<IntWritable> values = new ArrayList();values.add(new IntWritable(2));values.add(new IntWritable(3));//5 driver.withInput(new Text(key),values).withOutput(new Text("test"), new IntWritable(5)).runTest(); }

}

publicclass MapReduceTest {

private Reducer reducer; private Mapper mapper; private MapReduceDriver driver;@Before publicvoid init(){reducer = new WordCount.Reduce();mapper = new WordCount.Map();driver = new MapReduceDriver(mapper,reducer); } @Test publicvoid test() throws IOException{String line = "chinacache is a great CDN is it not";driver.withInput(new LongWritable(1),new Text(line)).withOutput(new Text("CDN"), new IntWritable(1)).withOutput(new Text("a"), new IntWritable(1)).withOutput(new Text("chinacache"), new IntWritable(1)).withOutput(new Text("great"), new IntWritable(1)).withOutput(new Text("is"), new IntWritable(2)).withOutput(new Text("it"), new IntWritable(1)).withOutput(new Text("not"), new IntWritable(1)).runTest(); }

}

總結(jié)

以上是生活随笔為你收集整理的使用MRUnit，Mockito和PowerMock进行Hadoop MapReduce作业的单元测试的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。