hive UDF函數取最新分區

1.pom文件

<dependencies><!-- https://mvnrepository.com/artifact/org.apache.hive/hive-exec --><dependency><groupId>org.apache.hive</groupId><artifactId>hive-exec</artifactId><version>1.2.1</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>2.7.6</version><exclusions><exclusion><groupId>log4j</groupId><artifactId>log4j</artifactId></exclusion><exclusion><groupId>org.slf4j</groupId><artifactId>slf4j-log4j12</artifactId></exclusion></exclusions></dependency><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.10</version><scope>test</scope></dependency></dependencies><build><pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) --><plugins><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-compiler-plugin</artifactId><version>3.0</version><configuration><source>1.8</source><target>1.8</target><encoding>UTF-8</encoding></configuration></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-shade-plugin</artifactId><version>2.2</version><executions><execution><phase>package</phase><goals><goal>shade</goal></goals><configuration><filters><filter><artifact>*:*</artifact><excludes><exclude>META-INF/*.SF</exclude><exclude>META-INF/*.DSA</exclude><exclude>META-INF/*/RSA</exclude></excludes></filter></filters></configuration></execution></executions></plugin></plugins></pluginManagement></build> </project>

2.代碼

import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hive.ql.exec.UDF;import java.net.URI; import java.util.ArrayList; import java.util.Collections; import java.util.List;/** * 該UDF函數取表最新分區，通過文件檢索以降低能耗 * 參數 tableName ：schema.table_name * 返回 latesttPatition：最新分區名稱 */public class latest_partition extends UDF {public String evaluate(String tableName) {StringBuffer sb = new StringBuffer();String latesttPatition = null;// 獲取shemaString split1 = tableName.split("\\.")[0];// 獲取table_nameString split2 = tableName.split("\\.")[1];// 拼接路徑String fileName = sb.append("/user/hive/warehouse/").append(split1).append(".db/").append(split2).toString();try{// 調用方法獲取最新分區latesttPatition = getFileList(fileName);}catch (Exception e){System.out.println("獲取結果異常" +e.getMessage());}return latesttPatition;}// 獲取最新分區public static String getFileList(String path) throws Exception{String res = null;Configuration conf=new Configuration(false);conf.set("fs.default.name", "hdfs://hacluster/");FileSystem hdfs = FileSystem.get(URI.create(path),conf);FileStatus[] fs = hdfs.listStatus(new Path(path));Path[] listPath = FileUtil.stat2Paths(fs);List<String> list = new ArrayList();for(Path p : listPath){String s = p.toString();// hdfs上有可能有非分區文件，只處理分區文件if(s.contains("=")) {String partition = s.split("=")[1];list.add(partition);}}if(list.size() != 0) {res = Collections.max(list).toString();}return res;}}

大表查詢最新分區往往由于各種原因，可能需要幾個小時，使用該函數可以實現秒級返回數據。性能可大范圍提升。

-- 優化前sql查詢語句(耗時特別久，全表掃描) SELECT MAX(dt) as latest_dt FROM table_name; -- 優化后(通過文件系統查詢，數秒返回結果) SELECT LST_DT('schema.table_name');

總結

以上是生活随笔為你收集整理的hive UDF函数取最新分区的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。