[YARN-2928] YARN Timeline Service v.2: alpha 1

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Component/s: timelineserver
• Hadoop Flags: Reviewed

Release Note

We are introducing an early preview (alpha 1) of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation.

YARN Timeline Service v.2 alpha 1 is provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. It should be used only in a test capacity. Most importantly, security is not enabled. Do not set up or use Timeline Service v.2 until security is implemented if security is a critical requirement.

More details are available in the [YARN Timeline Service v.2](./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html) documentation.

• Assignee: Sangjin Lee
• Reporter: Sangjin Lee
• Created: 05/Dec/14 22:52
• Updated: 25/Oct/19 20:24
• Resolved: 11/Jul/16 03:45

Description

We have the application timeline server implemented in yarn per YARN-1530 and YARN-321. Although it is a great feature, we have recognized several critical issues and features that need to be addressed.

This JIRA proposes the design and implementation changes to address those. This is phase 1 of this effort.

Issue Links

• incorporates: MAPREDUCE-6335 convert load test driver to timeline service v.2
• incorporates: MAPREDUCE-6337 add a mode to replay MR job history files to the timeline service
• incorporates: YARN-5236 FlowRunCoprocessor brings down HBase RegionServer
• incorporates: MAPREDUCE-6331 [Umbrella] Make MapReduce work with Timeline Service Nextgen (YARN-2928)
• is duplicated by: YARN-2177 Timeline server web interfaces high-availablity and scalability
• is duplicated by: YARN-2520 Scalable and High Available Timeline Server
• is duplicated by: YARN-2521 Reliable TimelineClient
• is related to: YARN-1530 [Umbrella] Store, manage and serve per-framework application-timeline data
• is related to: YARN-3538 TimelineServer doesn't catch/translate all exceptions raised
• is related to: YARN-5364 timelineservice modules have indirect dependencies on mapreduce artifacts
• is related to: YARN-5355 YARN Timeline Service v.2: alpha 2
• is related to: YARN-3332 [Umbrella] Unified Resource Statistics Collection per node
• is related to: YARN-321 [Umbrella] Generic application history service
• relates to: YARN-2517 Implement TimelineClientAsync
• relates to: YARN-2984 Metrics for container's actual memory usage
• relates to: YARN-2965 Enhance Node Managers to monitor and report the resource usage on machines
• relates to: MAPREDUCE-6732 mapreduce tasks for YARN Timeline Service v.2: alpha 2
• relates to: YARN-2141 [Umbrella] Capture container and node resource consumption
• relates to: HBASE-15686 Add override mechanism for the exempt classes when dynamically loading table coprocessor
• relates to: YARN-1012 Report NM aggregated container resource utilization in heartbeat

Sub-Tasks

• [Collector wireup] Set up timeline collector with basic request serving structure and lifecycle: Resolved
• [Data Model] create overall data objects of TS next gen: Resolved
• [Collector wireup] Implement RM starting its timeline collector: Resolved
• [Storage implementation] Create a test-only backing storage implementation for ATS writes: Resolved
• [Storage abstraction] Create backing storage write interface for timeline collectors: Resolved
• [Storage implementation] Create standalone HBase backing storage implementation for ATS writes: Resolved
• [Storage implementation] Create HBase cluster backing storage implementation for ATS writes: Resolved
• [Collector wireup] Implement timeline app-level collector service discovery: Resolved
• [Data Model] Make putEntities operation be aware of the app's context: Resolved
• [Data Model] Create ATS metrics API: Resolved
• [Data Model] Create ATS configuration, metadata, etc. as part of entities: Resolved
• [Event producers] Implement RM writing app lifecycle events to ATS: Resolved
• [Event producers] Implement NM writing container lifecycle events to ATS: Resolved
• [Data Serving] Set up ATS reader with basic request serving structure and lifecycle: Resolved
• [Source organization] Refactor timeline collector according to new code organization: Resolved
• [Data Serving] Handle how to set up and start/stop ATS reader instances: Resolved
• [Storage Implementation] Implement storage reader interface to fetch raw data from HBase backend: Resolved
• [Storage abstraction] Create backing storage read interface for ATS readers: Resolved
• [Data Serving] Provide a very simple POC html ATS UI: Resolved
• Bootstrap TimelineServer Next Gen Module: Resolved
• [Collector implementation] the REST server (web server) for per-node collector does not work if it runs inside node manager: Resolved
• [Collector wireup] We need an assured way to determine if a container is an AM container on NM: Resolved
• [Event producers] Change distributed shell to use new timeline service: Resolved
• [Storage implementation] Exploiting the option of using Phoenix to access HBase backend: Resolved
• [Documentation] Documenting the timeline service v2: Resolved
• [Collector implementation] Implement the core functionality of the timeline collector: Resolved
• [Data Mode] Implement client API to put generic entities: Resolved
• [Storage implementation] Create backing storage write interface and a POC only file based storage implementation: Resolved
• Refactor and fix null casting in some map cast for TimelineEntity (old and new) and fix findbug warnings: Resolved
• rename TimelineAggregator etc. to TimelineCollector: Resolved
• [Event Producers] NM TimelineClient container metrics posting to new timeline service.: Resolved
• Replace starting a separate thread for post entity with event loop in TimelineClient: Resolved
• Collector's web server should randomly bind an available port: Resolved
• TestTimelineServiceClientIntegration fails: Resolved
• Reuse TimelineCollectorManager for RM: Resolved
• Clearly define flow ID/ flow run / flow version in API and storage: Resolved
• Security support for new timeline service.: Resolved
• [Storage implementation] explore & create the native HBase schema for writes: Resolved
• Sub resources of timeline entity needs to be passed to a separate endpoint.: Resolved
• Cache runningApps in RMNode for getting running apps on given NodeId: Resolved
• Consolidate flow name/version/run defaults: Resolved
• Add miniHBase cluster and Phoenix support to ATS v2 unit tests: Resolved
• Consolidate data model change according to the backend implementation: Resolved
• unit tests failures and issues found from findbug from earlier ATS checkins: Resolved
• HttpServer2 Max threads in TimelineCollectorManager should be more than 10: Resolved
• RM only get back addresses of Collectors that NM needs to know.: Resolved
• Performance optimization using connection cache of Phoenix timeline writer: Resolved
• TestMRTimelineEventHandling and TestApplication are broken: Resolved
• Decide if flow version should be part of row key or column: Resolved
• Generalize native HBase writer for additional tables: Resolved
• build is broken on YARN-2928 branch due to possible dependency cycle: Resolved
• Fix TestHBaseTimelineWriterImpl unit test failure by fixing it's test data: Resolved
• Test case failures in TestDistributedShell and some issue fixes related to ATSV2: Resolved
• [JDK-8][YARN-2928] Exclude jdk.tools from hbase-client and hbase-testing-util: Resolved
• REST API implementation for getting raw entities in TimelineReader: Resolved
• [Aggregation] App-level aggregation and accumulation for YARN system metrics: Resolved
• add equals and hashCode to TimelineEntity and other classes in the data model: Resolved
• Support for fetching specific configs and metrics based on prefixes: Resolved
• Support complex filters in TimelineReader: Resolved
• Implement support for querying single app and all apps for a flow run: Resolved
• Add equals and hashCode to TimelineEntity: Resolved
• Populate flow run data in the flow_run & flow activity tables: Resolved
• Refactor timelineservice.storage to add support to online and offline aggregation writers: Resolved
• split the application table from the entity table: Resolved
• Bugs in HBaseTimelineWriterImpl: Resolved
• ensure timely flush of timeline writes: Resolved
• Fix new findbugs warnings in resourcemanager in YARN-2928 branch: Resolved
• Rethink event column key issue: Resolved
• Change to use the AM flag in ContainerContext determine AM container: Resolved
• Some of the NM events are not getting published due race condition when AM container finishes in NM: Resolved
• Change the way metric values are stored in HBase Storage: Resolved
• Publisher V2 should write the unmanaged AM flag and application priority: Resolved
• Deal with byte representations of Longs in writer code: Resolved
• Miscellaneous issues in NodeManager project: Resolved
• Add the flush and compaction functionality via coprocessors and scanners for flow run table: Resolved
• Populate the flow activity table: Resolved
• build is broken at TestHBaseTimelineWriterImpl.java: Resolved
• Support appUpdated event in TimelineV2 to publish details for movetoqueue, change in priority: Resolved
• [timeline reader] implement support for querying for flows and flow runs: Resolved
• [reader REST API] implement support for querying for flows and flow runs: Resolved
• Add a "skip existing table" mode for timeline schema creator: Resolved
• Refactor the SystemMetricPublisher in RM to better support newer events: Resolved
• Fix javadoc warnings floating up from hbase: Resolved
• [storage implementation] app id as string in row keys can cause incorrect ordering: Resolved
• [reader implementation] support flow activity queries based on time: Resolved
• Refactor reader classes in storage to nest under hbase specific package name: Resolved
• Add request/response logging & timing for each REST endpoint call: Resolved
• HBase reader throws NPE if Get returns no rows: Resolved
• Store user in app to flow table: Resolved
• Support fetching entities by UID and change the REST interface to conform to current REST APIs' in YARN: Resolved
• Support additional queries for ATSv2 Web UI: Resolved
• correctly set createdTime and remove modifiedTime when publishing entities: Resolved
• TestJobHistoryEventHandler and TestRMContainerAllocator failing on YARN-2928 branch: Resolved
• TestDistributedShell fails for V2 scenarios: Resolved
• ensure the timeline service v.2 is disabled cleanly and has no impact when it's turned off: Resolved
• Fix javadoc and checkstyle issues in timelineservice code: Resolved
• Unify the term flowId and flowName in timeline v2 codebase: Resolved
• Refactor reader API for better extensibility: Resolved
• Provide a mechanism to represent complex filters and parse them at the REST layer: Resolved
• TestTimelineAuthenticationFilter and TestYarnConfigurationFields fail: Resolved
• [Bug fix] RM fails to start when SMP is enabled: Resolved
• TestDistributedShell fails for v2 test cases after modifications for 1.5: Resolved
• TestRMRestart fails and findbugs issue in YARN-2928 branch: Resolved
• New findbugs warning in resourcemanager in YARN-2928 branch: Resolved
• ATS storage has one extra record each time the RM got restarted: Resolved
• NM is going down with NPE's due to single thread processing of events by Timeline client: Resolved
• CPU Usage Metric is not captured properly in YARN-2928: Resolved
• Add a check in the coprocessor for table to operated on: Resolved
• Ensure non-metric values are returned as is for flow run table from the coprocessor: Resolved
• Online aggregation logic should not run immediately after collectors got started: Resolved
• hbase unit tests fail due to dependency issues: Resolved
• Code cleanup for TestDistributedShell: Resolved
• [Documentation] Update timeline service v2 documentation to capture information about filters: Resolved
• upgrade HBase version for first merge: Resolved
• created time shows 0 in most REST output: Resolved
• flow activities and flow runs are populated with wrong timestamp when RM restarts w/ recovery enabled: Resolved
• timelinereader has a lot of logging that's not useful: Resolved
• NPE in Separator.joinEncoded(): Resolved
• timeline service build fails with java 8: Resolved
• entire time series is returned for YARN container system metrics (CPU and memory): Resolved
• timestamps are stored unencoded causing parse errors: Resolved
• YARN container system metrics are not aggregated to application: Resolved
• fix "no findbugs output file" error for hadoop-yarn-server-timelineservice-hbase-tests: Resolved
• fix findbugs warnings/errors for hadoop-yarn-server-timelineservice-hbase-tests: Resolved
• Escaping occurences of encodedValues: Resolved
• Eliminate singleton converters and static method access: Resolved
• [documentation] several updates/corrections to timeline service documentation: Resolved
• Make HBaseTimeline[Reader|Writer]Impl default and move FileSystemTimeline*Impl: Resolved
• NPE in Distributed Shell while publishing DS_CONTAINER_START event and other miscellaneous issues: Resolved
• Avoid re-creation of EventColumnNameConverter in HBaseTimelineWriterImpl#storeEvents: Resolved
• Eliminate unused imports checkstyle warnings: Resolved
• fix several rebase and other miscellaneous issues before merge: Resolved
• Store node information for finished containers in timeline v2: Resolved
• fix hadoop-aws pom not to do the exclusion: Resolved