[YARN-1530] [Umbrella] Store, manage and serve per-framework application-timeline data

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Unassigned
• Reporter: Vinod Kumar Vavilapalli
• Created: 23/Dec/13 23:29
• Updated: 14/Jun/15 11:40
• Resolved: 01/May/15 22:52

Description

This is a sibling JIRA for YARN-321.

Today, each application/framework has to do store, and serve per-framework data all by itself as YARN doesn't have a common solution. This JIRA attempts to solve the storage, management and serving of per-framework data from various applications, both running and finished. The aim is to change YARN to collect and store data in a generic manner with plugin points for frameworks to do their own thing w.r.t interpretation and serving.

Issue Links

• is depended upon by: MAPREDUCE-5858 [Umbrella] MR should make use of the timeline server
• is related to: YARN-1744 Renaming applicationhistoryservice module
• is related to: YARN-1701 Improve default paths of timeline store and generic history store
• is related to: SPARK-1537 Add integration with Yarn's Application Timeline Server
• is related to: YARN-1935 Security for timeline server
• is related to: YARN-321 [Umbrella] Generic application history service
• is related to: YARN-3539 Compatibility doc to state that ATS v1 is a stable REST API
• is related to: YARN-1452 Document the usage of the generic application history and the timeline data service
• relates to: HIVE-7076 Plugin (exec hook) to log to application timeline data to Yarn
• relates to: YARN-2928 YARN Timeline Service v.2: alpha 1

Sub-Tasks

• Define an in-memory implementation of ApplicationTimelineStore: Closed
• Define user-faced entity, entity-info and event objects: Closed
• Implement a Leveldb based ApplicationTimelineStore: Closed
• Implement timeline related web-services inside AHS for storing and retrieving entities+events: Closed
• Implement a client library for java users to post entities+events: Closed
• Add an integration test validating post, storage and retrival of entites+events: Resolved
• Define the ApplicationTimelineStore store as an abstraction for implementing different storage impls for storing timeline information: Closed
• Refactoring timeline classes to remove "app" related words: Closed
• Rethinking about POJO Classes: Resolved
• Sending timeline entities+events from Distributed shell: Closed
• Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore as default: Closed
• Replace MemoryApplicationTimelineStore with LeveldbApplicationTimelineStore as default: Resolved
• Review LICENSE and NOTICE to reflect new levelDB releated libraries being used: Closed
• Create an utility function to dump timeline records to json: Closed
• Enable offline deletion of entries in leveldb timeline store: Closed
• ATSWebServices produces jersey warnings: Closed
• TimelineWebServices always passes primary and secondary filters as strings: Closed
• Leveldb timeline store needs simple write locking: Closed
• Change types of related entities and primary filters in ATSEntity: Closed
• Improve logging in timeline service: Closed
• Timeline service getEntities API should provide ability to get entities from given id: Closed
• Make enabling timeline service configurable: Closed
• Improve default paths of timeline store and generic history store: Closed
• Timeline server is enabled by default: Closed
• LeveldbTimelineStore does not handle db exceptions properly: Closed
• Implement a scalable, available TimelineStore using HBase: Resolved
• Merging generic-history into the Timeline Store: Closed
• Rename the daemon name to timelineserver: Closed
• Rename internal names to being Timeline Service instead of application history: Resolved
• Leveldb timeline store applies secondary filters incorrectly: Resolved
• Add an admin module for the timeline server: Resolved
• DistributedShell publishes unfriendly user information to the timeline server: Closed
• Refactor timeline classes into server.timeline package: Closed
• Timeline server should validate the numeric configuration values: Closed
• Timelineserver should validate that yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms is greater than zero when level db is for timeline store: Resolved
• Timeline server web interfaces high-availablity and scalability: Resolved
• Add Cross-Origin support to the ATS REST API: Closed
• Move TimelineClient to yarn-common project: Closed
• Refactor TimelineWebServices: Closed
• Provide Cli to refesh Admin Acls for Timeline server: Resolved
• Allow enabling/disabling timeline server per framework: Closed
• Rethinking of the HTTP method of TimelineWebServices#postEntities: Resolved
• TimelineClient should wrap all GET APIs to facilitate Java users: Resolved
• Document Cross Origin Filter Configuration for ATS: Closed
• Cross Origin configuration parameters prefix are not honored: Closed
• Enable Cross Origin Filter for timeline server only and not all Yarn servers: Closed
• Allow All Origins by default when Cross Origin Filter is enabled: Closed
• Allow for origin pattern matching in cross origin filter: Closed
• Host framework UIs in YARN for use with the ATS: Closed
• Implement TimelineClientAsync: Resolved
• Scalable and High Available Timeline Server: Resolved
• Reliable TimelineClient: Resolved
• Cross Origin Filter Http response split vulnerability protection rejects valid origins: Closed
• Cross Origin Filter throws UnsupportedOperationException upon destroy: Closed
• Tool to measure the performance of the timeline server: Resolved
• Document of timeline server needs to be updated: Resolved
• Add retry for timeline client put APIs: Closed
• Add retry for timeline client getDelegationToken method: Closed
• RM uses conf instead of token service address to renew timeline delegation tokens: Closed
• Add Rolling Time To Lives Level DB Plugin Capabilities: Resolved
• Add version info on timeline service / generic history web UI and REST API: Resolved