[YARN-321] [Umbrella] Generic application history service

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Assignee: Unassigned
• Reporter: Luke Lu
• Created: 21/Sep/11 17:10
• Updated: 27/Oct/16 19:39
• Resolved: 01/May/15 22:15

Description

The mapreduce job history server currently needs to be deployed as a trusted server in sync with the mapreduce runtime. Every new application would need a similar application history server. Having to deploy O(T*V) (where T is number of type of application, V is number of version of application) trusted servers is clearly not scalable.

Job history storage handling itself is pretty generic: move the logs and history data into a particular directory for later serving. Job history data is already stored as json (or binary avro). I propose that we create only one trusted application history server, which can have a generic UI (display json as a tree of strings) as well. Specific application/version can deploy untrusted webapps (a la AMs) to query the application history server and interpret the json for its specific UI and/or analytics.

Issue Links

• contains: YARN-1621 Add CLI to list rows of <task attempt ID, container ID, host of container, state of container>
• is duplicated by: YARN-304 RM Tracking Links for purged applications needs a long-term solution
• is related to: YARN-1701 Improve default paths of timeline store and generic history store
• is related to: YARN-1982 Rename the daemon name to timelineserver
• is related to: YARN-374 Job History Server doesn't show jobs which killed by ClientRMProtocol.forceKillApplication
• is related to: YARN-1452 Document the usage of the generic application history and the timeline data service
• relates to: YARN-374 Job History Server doesn't show jobs which killed by ClientRMProtocol.forceKillApplication
• relates to: YARN-1530 [Umbrella] Store, manage and serve per-framework application-timeline data
• relates to: YARN-2928 YARN Timeline Service v.2: alpha 1

Sub-Tasks

• HistoryStorage Reader Interface for Application History Server: Closed
• Bootstrap ApplicationHistoryService module: Closed
• HistoryStorage writer interface for Application History Server: Closed
• YARN-321 branch is broken due to applicationhistoryserver module's pom.xml: Closed
• Adding HDFS implementation for History Reader Interface: Resolved
• Defining the history data classes for the implementation of the reading/writing interface: Closed
• [YARN-321] Enable ResourceManager to write history data: Closed
• [YARN-321] History Service should create the webUI and wire it to HistoryStorage: Closed
• [YARN-321] Implementation of ApplicationHistoryProtocol: Closed
• [YARN-321] Add a testable in-memory HistoryStorage: Closed
• Update application_history_service.proto: Closed
• [YARN-321] Command Line Interface(CLI) for Reading Application History Storage Data: Closed
• RMContainer should collect more useful information to be recorded in Application-History: Closed
• Add a file-system implementation for history-storage: Closed
• [YARN-321] Adding ApplicationAttemptReport and Protobuf implementation: Closed
• [YARN-321] Add more APIs related to ApplicationAttempt and Container in ApplicationHistoryProtocol: Closed
• [YARN-321] Move classes from applicationhistoryservice.records.pb.impl package to applicationhistoryservice.records.impl.pb: Closed
• Adding ApplicationHistoryManager responsible for exposing reports to all clients: Closed
• Optimizing the reading/writing operations of FileSystemHistoryStorage: Resolved
• [YARN-321] Enhance History Reader interface for Containers: Closed
• [YARN-321] Webservices REST API's support for Application History: Closed
• Adding AHS as service of RM: Resolved
• Improve toString implementation for PBImpls for AHS: Resolved
• [YARN-321] Adding ContainerReport and Protobuf implementation: Closed
• [YARN-321] Update artifact versions for application history service: Closed
• Script changes to start AHS as an individual process: Closed
• Generic history service should support application-acls: Closed
• Implement PB service and client wrappers for ApplicationHistoryProtocol: Closed
• Add AHSDelegationTokenSecretManager for ApplicationHistoryProtocol: Resolved
• AHS History Store Cache Implementation: Resolved
• Separate ApplicationAttemptStartDataProto and ApplicationAttemptRegisteredDataProto: Resolved
• Removing FINAL_SAVING from YarnApplicationAttemptState: Closed
• Revisit the output type of the reader interface: Resolved
• Limit the number of outstanding tfile writers in FileSystemApplicationHistoryStore: Resolved
• [YARN-321] AHS protocols need to be in yarn proto package name after YARN-1170: Closed
• ApplicationClientProtocol and ApplicationHistoryProtocol should expose analogous APIs: Closed
• [YARN-321] AHS WebUI should server aggregated logs as well: Resolved
• RM cannot write the final state of RMApp/RMAppAttempt to the application history store in the transition to the final state: Closed
• Make aggregated logs of completed containers available via REST API: Resolved
• RM Tracking Links for purged applications needs a long-term solution: Resolved
• [YARN-321] Failing tests in org.apache.hadoop.yarn.server.applicationhistoryservice.*: Closed
• [YARN-321] Merge Patch for YARN-321: Closed
• YARN-321 branch needs to be updated after YARN-888 pom changes: Closed
• Test failures on YARN-321 branch: Closed
• Javadoc failures on YARN-321 branch: Closed
• FindBugs warnings on YARN-321 branch: Closed
• Fix formatting issues with new module in YARN-321 branch: Closed
• TestAHSWebApp failed in YARN-321 branch: Closed
• Fix how to read history file in FileSystemApplicationHistoryStore: Closed
• Fix config name YARN_HISTORY_SERVICE_ENABLED: Closed
• mvn apache-rat:check outputs warning message in YARN-321 branch: Closed
• Fix history server heap size in yarn script: Closed
• Bugs around log URL: Closed
• AHS records non-launched containers: Resolved
• Renaming applicationhistoryservice module: Resolved
• Review AHS configs and sync them up with the timeline-service configs: Closed
• yarn applicationattempt/container print wrong usage information: Closed
• Yarn CLI only shows running containers for Running Applications: Resolved
• ApplicationHistoryClientService#getApplications needs to be able to filter applications: Open
• YarnClient will not be redirected to the history server when RM is done: Resolved
• History client service needs to be more robust: Resolved
• Uniform the XXXXNotFound messages from ClientRMService and ApplicationHistoryClientService: Closed
• FileSystemApplicationHistoryStore blocks RM and AHS while NN is in safemode: Closed
• Removing old application history store after we store the history data to timeline store: Resolved
• Few fields displaying wrong values in Timeline server after RM restart: Resolved
• Jobs are not displaying in timeline server after RM restart: Resolved
• Investigating whether generic history service needs to support queue-acls: Resolved
• Augment HistoryStorage Reader Interface to Support Filters When Getting Applications: Resolved
• FileSystemApplicationHistoryStore#HistoryFileReader#next() should check return value of dis.read(): Resolved
• AHSClient may be not necessary: Resolved
• Generic history service RPC interface doesn't work when service authorization is enabled: Closed
• "yarn application -status <appId>" throws NPE when retrieving the app from the timelineserver: Closed
• Resource usage should be published to the timeline server as well: Closed
• AHSWebServices should return FORBIDDEN(403) if the request user doesn't have access to the history data: Closed
• GHS should show N/A instead of null for the inaccessible information: Resolved
• ApplicationHistoryManager is expected to return a sorted list of apps/attempts/containers: Closed
• Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500): Closed