[YARN-4233] YARN Timeline Service plugin: ATS v1.5

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: timelineserver
• Assignee: Xuan Gong
• Reporter: Xuan Gong
• Created: 07/Oct/15 17:14
• Updated: 04/Nov/16 01:06

Description

Copy the description from YARN-3942: Timeline store to read events from HDFS
This adds a new timeline store plugin that is intended as a stop-gap measure to mitigate some of the issues we've seen with ATS v1 while waiting for ATS v2. The intent of this plugin is to provide a workable solution for running the Tez UI against the timeline server on a large-scale clusters running many thousands of jobs per day.

Issue Links

• depends upon: YARN-4772 Overloaded leveljb can crash the ATS "pthread lock: Invalid argument"
• is related to: YARN-4705 ATS 1.5 parse pipeline to consider handling open() events recoverably
• is related to: YARN-4774 ATS can block in shutdown under heavy load
• is related to: HADOOP-12906 AuthenticatedURL should convert a 404/Not Found into an FileNotFoundException.
• relates to: YARN-5835 NoClassDefFoundError when reading ATS v1.5

Sub-Tasks

• Timeline store to read events from HDFS: Resolved
• New levelDB cache storage for timeline v1.5: Resolved
• New put APIs in TimelineClient for ats v1.5: Resolved
• Provide new timeline plugin storage to support fine-grained entity caching: Resolved
• Document new put APIs in TimelineClient for ATS 1.5: Resolved
• Allow YARN distributed shell to use ATS v1.5 APIs: Resolved
• TimerTasks leak in ATS V1.5 Writer: Resolved
• Allow YARN system metrics publisher to use ATS v1.5 APIs: Open
• Document ATS v1.5: Open
• EntityGroupFSTimelineStore to not log errors during shutdown: Resolved
• Improving EntityGroupFSTimelineStore on exception handling, test setup, and concurrency: Resolved
• ATS 1.5 parse pipeline to consider handling open() events recoverably: Open
• TimelineClient to implement Flushable; propagate to writer: Resolved
• ATS 1.5 timelineclient impl call flush after every event write: Resolved
• ATS 1.5 timelineclient impl try to create attempt directory for every event call: Resolved
• Change Log Level to DEBUG for putDomain call in ATS 1.5: Resolved
• Metric improvements for ATS v1.5 storage components: Resolved
• Add HDFS caller context for EntityGroupFSTimelineStore: Resolved
• EntityGroupFSTimelineStore to close cached leveldb instances in serviceStop(): Resolved
• findbug issue on LevelDBCacheTimelineStore: Resolved
• Read cache concurrency issue between read and evict in EntityGroupFS timeline store: Resolved
• Support for specifying a path for ATS plugin jars: Resolved
• Support for specifying a path for ATS plugin jars: Resolved
• EntityGroupFSTimelineStore should allow configurable cache stores.: Resolved