[YARN-4696] Improving EntityGroupFSTimelineStore on exception handling, test setup, and concurrency

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: timelineserver
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 16/Feb/16 14:23
• Updated: 30/Aug/16 01:06
• Resolved: 10/Mar/16 19:02

Description

EntityGroupFSTimelineStore now depends on an RM being up and running; the configuration pointing to it. This is a new change, and impacts testing where you have historically been able to test without an RM running.

The sole purpose of the probe is to automatically determine if an app is running; it falls back to "unknown" if not. If the RM connection was optional, the "unknown" codepath could be called directly, relying on age of file as a metric of completion

Options

1. add a flag to disable RM connect
2. skip automatically if RM not defined/set to 0.0.0.0
3. disable retries on yarn client IPC; if it fails, tag app as unknown.

Issue Links

• contains: YARN-4772 Overloaded leveljb can crash the ATS "pthread lock: Invalid argument"
• incorporates: YARN-4695 EntityGroupFSTimelineStore to not log errors during shutdown
• incorporates: YARN-4716 TimelineClient to implement Flushable; propagate to writer
• is related to: YARN-4705 ATS 1.5 parse pipeline to consider handling open() events recoverably