[YARN-10975] EntityGroupFSTimelineStore#ActiveLogParser parses already processed files - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.3.0
Fix Version/s: 3.4.0
Component/s: timelineserver
Labels:
- pull-request-available

Description

EntityGroupFSTimelineStore#ActiveLogParser parses already processed files again and again even though there is no change in the file. This leads to unnecessary load on DFS where summary files resides and Timeline Store where timeline entities present.

2021-10-10 19:20:43,940 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 275 msec
2021-10-10 19:21:44,079 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 341 msec
2021-10-10 19:22:44,065 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 335 msec
2021-10-10 19:23:44,038 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 370 msec
2021-10-10 19:24:44,087 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 317 msec
2021-10-10 19:25:44,092 INFO  timeline.LogInfo - Parsed 6 entities from hdfs:/prabhuJoseph/atshistory/active/application_1631559260564_0009/appattempt_1631559260564_0009_000001/summarylog-appattempt_1631559260564_0009_000001_2331123893 in 336 msec