[YARN-10975] EntityGroupFSTimelineStore#ActiveLogParser parses already processed files

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0
• Component/s: timelineserver
• Labels: pull-request-available
• Assignee: Ravuri Sushma sree
• Reporter: Prabhu Joseph
• Created: 13/Oct/21 12:16
• Updated: 24/Jul/23 06:30
• Resolved: 29/Nov/21 18:31

Description

EntityGroupFSTimelineStore#ActiveLogParser parses already processed files again and again even though there is no change in the file. This leads to unnecessary load on DFS where summary files resides and Timeline Store where timeline entities present.

2021-10-10 19:20:43,940 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 275 msec
2021-10-10 19:21:44,079 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 341 msec
2021-10-10 19:22:44,065 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 335 msec
2021-10-10 19:23:44,038 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 370 msec
2021-10-10 19:24:44,087 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 317 msec
2021-10-10 19:25:44,092 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 336 msec

Issue Links

• links to: GitHub Pull Request #3735