[YARN-11170] EntityGroupFSTimelineStore cleaner thread stops if it fails with error

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.10.0
• Component/s: timelineserver
• Assignee: Unassigned
• Reporter: Akash
• Created: 03/Jun/22 07:39
• Updated: 03/Jun/22 07:48

Description

timeline server v1.5 hdfs data cleanup thread stops cleaning when it finds a file missing 

 

2022-06-01 08:00:25,861 ERROR org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore: Error cleaning files
java.io.FileNotFoundException: File hdfs://<namenode-nameservice>/ats-cluster2/entity-file-history/done/1646401932991/0000/650/application_1646401932991_650010 does not exist.
        at org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1144)

 

We found that older files are present 

 

drwxrwxr-x   - hadoop supergroup          0 2022-03-16 01:40 hdfs://<namenode-nameservice>/ats-cluster2/entity-file-history/done/1646401932991
drwxrwxr-x   - hadoop supergroup          0 2022-05-17 19:43 hdfs://<namenode-nameservice>/ats-cluster2/entity-file-history/done/1648056692622

our configs are 

yarn.timeline-service.ttl-enable=ture
yarn.timeline-service.client.internal-timers-ttl-secs=420
yarn.timeline-service.ttl-ms=432000000

This thread should ignore the error and should continue cleaning up further directories