Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.10.0
-
None
-
None
Description
timeline server v1.5 hdfs data cleanup thread stops cleaning when it finds a file missing
2022-06-01 08:00:25,861 ERROR org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore: Error cleaning files
java.io.FileNotFoundException: File hdfs://<namenode-nameservice>/ats-cluster2/entity-file-history/done/1646401932991/0000/650/application_1646401932991_650010 does not exist.
at org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1144)
We found that older files are present
drwxrwxr-x - hadoop supergroup 0 2022-03-16 01:40 hdfs://<namenode-nameservice>/ats-cluster2/entity-file-history/done/1646401932991 drwxrwxr-x - hadoop supergroup 0 2022-05-17 19:43 hdfs://<namenode-nameservice>/ats-cluster2/entity-file-history/done/1648056692622
our configs are
yarn.timeline-service.ttl-enable=ture yarn.timeline-service.client.internal-timers-ttl-secs=420 yarn.timeline-service.ttl-ms=432000000
This thread should ignore the error and should continue cleaning up further directories