[YARN-4063] Populate the flow activity table - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Convert to Issue

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Resolved
Priority: Major
Resolution: Duplicate
Affects Version/s: None
Fix Version/s: None
Component/s: timelineserver
Labels:
None

Release Note:

Hide
The code for flow activity table creation, population and testing is part of the patch at ~~YARN-3901~~. Hence marking this as duplicate.

Dealing with long running apps is part of https://issues.apache.org/jira/browse/YARN-4069

Show
The code for flow activity table creation, population and testing is part of the patch at YARN-3901 . Hence marking this as duplicate. Dealing with long running apps is part of https://issues.apache.org/jira/browse/YARN-4069

Description

Need to populate the flow_activity table

-Stores per day flow run pointers and info
-Written to by RM’s collector for application lifecycle
primary key: cluster ! day timestamp ! user ! flow id
-For the day timestamp we can take the millis since epoch for the end of the day (24:00h).
columns include runids, start time, end time, snapshot time
-This table will also be used to efficiently retrieve the flows that had an activity in a certain day. That is needed for daily aggregations, but also for several UIs, including a flow-based UI.