Could not close the output stream for file hdfs://192.168.190.129:9000/BJ_4.c
在使用kettle從windows客戶端向hadoop集群傳輸文件的過程中,如下:
報這樣的錯誤,文件上傳成功,但是沒有寫入數據。
2017/04/12 15:32:08 - Hadoop Copy Files - 正在處理行, 源文件/目錄: [E:/ArcGIS/2009-05-16_BJ_4.csv] ... 目標文件/目錄 : [hdfs://192.168.190.129:9000/nytaxi/csv]... 通配符 : [ ^.*\.csv]
2017/04/12 15:32:08 - Hadoop Copy Files - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : 文件系統異常:Could not copy "file:///E:/ArcGIS/2009-05-16_BJ_4.csv" to "hdfs://192.168.190.129:9000/nytaxi/csv/2009-05-16_BJ_4.csv".2017/04/12 15:32:08 - Hadoop Copy Files - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : 原因:Could not close the output stream for file "hdfs://192.168.190.129:9000/nytaxi/csv/2009-05-16_BJ_4.csv".
2017/04/12 15:32:08 - Hadoop Copy Files - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : 原因:DataStreamer Exception:
2017/04/12 15:32:08 - Hadoop Copy Files - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : 原因:Could not initialize class org.apache.hadoop.hdfs.protocol.HdfsConstants
2017/04/12 15:32:08 - job_local2hdfs - 完成作業項[Hadoop Copy Files] (結果=[false])
2017/04/12 15:32:08 - job_local2hdfs - 任務執行完畢
2017/04/12 15:32:08 - Spoon - 任務已經結束.
自己百度折騰了好久,一直不知道什么原因,最后在官網看到:
Problem: Hadoop copy files step creates an empty file in HDFS and hangs or never writes any data.
Check: The Hadoop client side API that Pentaho calls to copy files to HDFS requires that PDI has network connectivity to the nodes in the cluster. The DNS names or IP addresses used within the cluster must resolve the same relative to the PDI machine as they do in the cluster. When PDI requests to put a file into HDFS, the Name Node will return the DNS names (or IP address' depending on the configuration) of the actual nodes that the data will be copied to.
大概意思是網絡問題,不在同一個網段內。解決辦法是把kettle安裝在linux上,再用在xmanager中的xshell運行進入圖形化界面
參考:http://www.mamicode.com/info-detail-1458941.html
那么問題來了,本來想從windows直接上傳文件到hdfs上,跳過中間上傳到linux這個環節,但是kettle部署在linux上,那不是要把文件還得上傳到linux上,這樣做沒有意義了。
總結
以上是生活随笔為你收集整理的Could not close the output stream for file hdfs://192.168.190.129:9000/BJ_4.c的全部內容,希望文章能夠幫你解決所遇到的問題。
- 上一篇: ArcGIS API for Pytho
- 下一篇: kettle连接hadoophdfs图文