flume打包遇到的一些問題

1.ipc兼容性問題，線上使用2.3.0的hdfs，但是打包時默認為1.2.1的

1 2 3 4

08?Apr?2015?19:38:25,122?WARN??[SinkRunner-PollingRunner-DefaultSinkProcessor]?(org.apache.flume.sink.hdfs.HDFSEventSink.process:455)??-?HDFS?IO?error org.apache.hadoop.ipc.RemoteException:?Server?IPC?version?9?cannot?communicate?with?client?version?4 ????????at?org.apache.hadoop.ipc.Client.call(Client.java:1113) ????????at?org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)

拷貝hadoop-core包即可,或者更改pom.xml文件

1 2 3 4 5 6

??????<properties> ????????<hadoop.version>1.2.1</hadoop.version>?//hadoop-core-1.2.1.jar??改為hadoop-core-2.3.0-mr1-cdh5.1.0.jar ????????<hbase.version>0.92.1</hbase.version> ????????<hadoop.common.artifact.id>hadoop-core</hadoop.common.artifact.id> ????????<thrift.version>0.7.0</thrift.version> ??????</properties>

或者打包時指定：

1	mvn?clean?install?-Phadoop-2

2.沒有跳過test時

1	org.apache.flume.auth.TestFlumeAuthenticator:?org/apache/commons/io/Charsets?(no?class?error)

根目錄下的pom.xml文件中更改為:

1 2 3 4 5

<dependency> ???<groupId>commons-io</groupId> ???<artifactId>commons-io</artifactId> ???<version>2.4</version>?//2.1的包沒有Charsets這個類 </dependency>

如果手動下載jar包可以使用如下命令導(dǎo)入：

1	mvn?install:install-file?-DgroupId=commons-io?-DartifactId=commons-io?-Dversion=2.0.1?-Dpackaging=jar?-Dfile=commons-io-2.4.jar

3. 程序語言問題，因為在系統(tǒng)中使用了中文環(huán)境導(dǎo)致編譯不通過

1）

1 2 3 4 5 6

@Test ??public?void?shouldUseUtcAsBasisForDateFormat()?{ ????assertEquals("Coordinated?Universal?Time", ????????factory.fastDateFormat.getTimeZone().getDisplayName());?//請求英文，返回中文 ??} ...

2）

1 2 3 4 5 6

Running?org.apache.flume.source.twitter.TestTwitterSource Tests?run:?2,?Failures:?0,?Errors:?1,?Skipped:?1,?Time?elapsed:?0.247?sec?<<<?FAILURE! testCarrotDateFormatBug(org.apache.flume.source.twitter.TestTwitterSource)??Time?elapsed:?21?sec??<<<?ERROR! java.text.ParseException:?Unparseable?date:?"Fri?Oct?26?22:53:55?+0000?2012" ???at?java.text.DateFormat.parse(DateFormat.java:357) ???at?org.apache.flume.source.twitter.TestTwitterSource.testCarrotDateFormatBug(TestTwitterSource.java:109)

代碼：

1 2 3 4 5

@Test ??public?void?testCarrotDateFormatBug()?throws?Exception?{ ????SimpleDateFormat?formatterFrom?=?new?SimpleDateFormat("EEE?MMM?dd?HH:mm:ss?Z?yyyy");?//更改為new?SimpleDateFormat("EEE?MMM?dd?HH:mm:ss?Z?yyyy",Locale.US); ????formatterFrom.parse("Fri?Oct?26?22:53:55?+0000?2012"); ??}

或者簡單點，更改語言設(shè)置或者直接skip test

1	mvn?clean?install?-Phadoop-2?-DskipTests

4.json問題

替換單個jar包時，因為依賴問題，包class not found:

1 2 3 4 5

Exception?in?thread?"PollableSourceRunner-KafkaSource-kafka1"?java.lang.NoClassDefFoundError:?net/sf/ezmorph/Morpher ????????at?org.apache.flume.source.kafka.KafkaSourceUtil.getDateMessage(KafkaSourceUtil.java:117) ????????at?org.apache.flume.source.kafka.KafkaSource.process(KafkaSource.java:123) ????????at?org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139) ????????at?java.lang.Thread.run(Thread.java:745)

可以打成ensmbly包

1 2 3 4 5 6 7

flume-1.6.0/flume-ng-sources/flume-kafka-source/pom.xml <dependency> ???<groupId>net.sf.json-lib</groupId> ???<artifactId>json-lib</artifactId> ???<version>2.4</version> ???<classifier>jdk15</classifier> </dependency>

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

<plugin> ????<artifactId>maven-assembly-plugin</artifactId> ????<version>2.4</version> ????<configuration> ??????<descriptorRefs> ????????<descriptorRef>jar-with-dependencies</descriptorRef> ??????</descriptorRefs> ????</configuration> ????<executions> ??????<execution> ????????<id>make-assembly</id> ????????<phase>package</phase> ????????<goals> ??????????<goal>single</goal> ????????</goals> ??????</execution> ????</executions> ??</plugin>

或者拷貝jar包：

1 2 3 4 5 6

/Users/nizengguang/.m2/repository/net/sf/ezmorph/ezmorph/1.0.6/ezmorph-1.0.6.jar /Users/nizengguang/.m2/repository/net/sf/json-lib/json-lib/2.4/json-lib-2.4-jdk15.jar /Users/nizengguang/.m2/repository/commons-beanutils/commons-beanutils/1.8.0/commons-beanutils-1.8.0.jar /Users/nizengguang/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar /Users/nizengguang/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar /Users/nizengguang/.m2/repository/commons-lang/commons-lang/2.5/commons-lang-2.5.jar

本文轉(zhuǎn)自菜菜光 51CTO博客，原文鏈接：http://blog.51cto.com/caiguangguang/1635779，如需轉(zhuǎn)載請自行聯(lián)系原作者 創(chuàng)作挑戰(zhàn)賽新人創(chuàng)作獎勵來咯，堅持創(chuàng)作打卡瓜分現(xiàn)金大獎

總結(jié)

以上是生活随笔為你收集整理的flume package遇到的问题的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。