[YARN-11093] In fs-support-append false case, timelineserver reads event files in arbitrary order

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 3.3.2
• Component/s: timelineserver
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: shailesh gupta
• Created: 22/Mar/22 12:27
• Updated: 25/Dec/23 09:18

Description

In our setup, we are using Hive and Tez and using offline tez-ui mode, where we copy the ATS events files to other place and start tez-ui. To keep the event files small, we are using 

fs-support-append = false, and creating a new file for writing the events. We can see that, in this mode, file name contains suffix as timestamp.

But at the time of read, we are relying on the FS to provide correct file order. If that fails, we are adding event out of order, leading to event being discarded, or incorrect information.

Fix could be sorting of the file names, based on suffix if append mode is not used.

sample file names:

• summarylog-appattempt_1647348120288_0001_000001_460237
• entitylog-timelineEntityGroupId_1647348120288_1_dag_1647348120288_0001_1_673147

 

Issue Links

• links to: GitHub Pull Request #4093