Is there a way to delete files older than 10 days on HDFS?

In Linux I would use:

find /path/to/directory/ -type f -mtime +10 -name '*.txt' -execdir rm -- {} \;

Is there a way to do this on HDFS? (Deletion to be done based on file creation date)

解決方案

Solution 1: Using multiple commands as answered by daemon12

hdfs dfs -ls /file/Path | tr -s " " | cut -d' ' -f6-8 | grep "^[0-9]" | awk 'BEGIN{ MIN=14400; LAST=60*MIN; "date +%s" | getline NOW } { cmd="date -d'\''"$1" "$2"'\'' +%s"; cmd | getline WHEN; DIFF=NOW-WHEN; if(DIFF > LAST){ print "Deleting: "$3; system("hdfs dfs -rm -r "$3) }}'

Solution 2:

Using Shell script

today=`date +'%s'`

hdfs dfs -ls /file/Path/ | grep "^d" | while read line ; do

dir_date=$(echo ${line} | awk '{print $6}')

difference=$(( ( ${today} - $(date -d ${dir_date} +%s) ) / ( 24*60*60 ) ))

filePath=$(echo ${line} | awk '{print $8}')

if [ ${difference} -gt 10 ]; then

hdfs dfs -rm -r $filePath

fi

done

總結(jié)

以上是生活随笔為你收集整理的shell 删除了hdfs 文件,在HDFS上删除超过10天的文件的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。