[Linux]结合awk删除hdfs指定日期前的数据
生活随笔
收集整理的這篇文章主要介紹了
[Linux]结合awk删除hdfs指定日期前的数据
小編覺得挺不錯的,現在分享給大家,幫大家做個參考.
業務背景
約定五天前的HDFS數據為過期版本號數據。寫一個腳本自己主動刪除過期版本號數據
$ hadoop fs -ls /user/pms/workspace/ouyangyewei/data Found 9 items drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-01 drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-02 drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-03 drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-04 drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-05 drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-06 drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-07 drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-08 drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-09腳本實現
# --------------------------------------------------------- # # 刪除歷史版本號(五天前的為過期版本號數據) # # ---------------------------------------------------------old_version=$(hadoop fs -ls /user/pms/workspace/ouyangyewei/data | awk 'BEGIN{ five_days_ago=strftime("%F", systime()-5*24*3600) }{ split($8,arr,"/"); if(arr[7]<five_days_ago){printf "%s\n", $8} }') arr=(${old_version// / }) for version in ${arr[@]} do hadoop fs -rmr $version done運行以后
$ hadoop fs -ls /user/pms/workspace/ouyangyewei/data Found 4 items drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-06 drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-07 drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-08 drwxr-xr-x - pms pms 0 2015-08-11 17:03 /user/pms/workspace/ouyangyewei/data/2015-08-09轉載于:https://www.cnblogs.com/jzdwajue/p/7218361.html
總結
以上是生活随笔為你收集整理的[Linux]结合awk删除hdfs指定日期前的数据的全部內容,希望文章能夠幫你解決所遇到的問題。
- 上一篇: 如何简单的测试kubernetes的dn
- 下一篇: cobbler自动化安装Linux系统