[YARN-7055] YARN Timeline Service v.2: beta 1 / GA

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.2.1
• Component/s: timelineclient, timelinereader, timelineserver
• Release Note: Application Timeline Server v2 is ready for production. It is GA from 3.2.1 release on wards.
• Assignee: Unassigned
• Reporter: Vrushali C
• Created: 19/Aug/17 05:22
• Updated: 23/Oct/19 16:15
• Resolved: 05/Sep/19 13:37

Description

This is an umbrella JIRA for the beta 1 milestone for YARN Timeline Service v.2.

YARN-2928 was alpha1, YARN-5355 was alpha2.

Issue Links

• incorporates: YARN-7213 [Umbrella] Test and validate HBase-2.0.x with Atsv2
• is related to: YARN-5355 YARN Timeline Service v.2: alpha 2

Sub-Tasks

• Limit the flow name size & consider cleanup for hex chars: Resolved
• Ship single node HBase config option with single startup command: Resolved
• Explore supporting a simpler back-end implementation for ATS v2: Resolved
• [Collector wireup] Implement NM starting the standalone timeline collector daemon: Resolved
• [Collector wireup] Handle timeline collector failure scenarios: Resolved
• [Collector wireup] Work-preserving restarting of per-node timeline collector: Resolved
• [Data Serving] Clustering of ATS reader instances: Resolved
• [Source organization] Decide detailed package structures for timeline service v2 components: Resolved
• [Compatibility] Make TS next gen be compatible with the current TS: Resolved
• [Event producers] Change distributed shell to use TIMELINE_SERVICE_VERSION: Resolved
• provide RPC metrics via JMX for timeline collectors and readers: Resolved
• CollectorNodemanagerProtocol's authorization doesn't work: Resolved
• Investigate the concurrency issue with the map of timeline collector: Resolved
• determine how to generate YARN container events: Resolved
• Enable application client to communicate with new timeline service: Resolved
• Consider concurrency situations for TimelineWriter: Resolved
• consider nulling member maps/sets of TimelineEntity: Resolved
• [Aggregation] Application/Flow/User/Queue Level Aggregations: Resolved
• [Aggregation] Flow and User level aggregation on Application States table: Resolved
• [Collector implementation] Implement timeline collector functionality to serve ATS readers' requests for live apps: Resolved
• [Aggregation] Queue-level Aggregation on Application States table: Resolved
• TimelineReader Web UI Implementation: Resolved
• Flow page for Web UI: Resolved
• some YARN container events have timestamp of -1: Resolved
• Serve aggregated logs of historical apps from timeline service: Resolved
• [Storage implementation] Create HDFS backing storage implementation for ATS reads: Resolved
• Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a consistent way: Resolved
• [ATSv2] : NPE for recovered finished applications in NM during upgrade: Resolved
• Make the aggregation interval in AppLevelTimelineCollector configurable: Resolved
• [ATSv2] Fix findbugs warnings: Resolved
• Consider writing to both ats v1 & v2 from RM for smoother upgrades: Resolved
• Have a way to turn off container metrics from NMs: Resolved
• Update ATSv2 document for latest feature supports: Resolved
• [Storage implementation] Adding retry semantics to HDFS backing storage: Resolved
• Allow port range to be specified while starting NM Timeline collector manager.: Resolved
• start time key in NM leveldb store should be removed when container is removed: Resolved
• Fix findbugs warning in timeline service on branch-2: Resolved
• The error message in TimelineSchemaCreator is not enough to find out the error.: Resolved
• Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2): Resolved
• TestTimelineReaderWebServicesHBaseStorage fails with NoClassDefFoundError on TSv2 branch2: Resolved
• Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath: Resolved
• Backport ATSv2 to branch-2: Resolved
• [atsv2 read acls] Include domain table creation as part of schema creator: Resolved
• Documentation changes post branch-2 merge: Resolved
• Update LICENSE.txt with respect to new YARN UI merge: Resolved
• [Atsv2] load hbase configuration from filesystem rather than URL: Resolved
• YARN_TIMELINESERVER_OPTS is not valid env variable for TimelineReaderServer: Resolved
• metricsTimeStart and metricsTimeEnd should be all lower case in the doc: Resolved
• [Atsv2] Update document for running v1 and v2 TS: Resolved
• FlowRunReader constructs min_start_time filter for both createdtimestart and createdtimeend.: Resolved
• [atsv2 read acls] Add TimelineWriter#writeDomain: Resolved
• [Atsv2] Retrospect storing entities into sub application table from client perspective: Resolved
• Add table for storing timeline domain information: Resolved
• ATSv2 REST API queries do not return results for uppercase application tags: Resolved
• [atsv2 read acls] REST API to get domain info by id: Resolved
• YarnClient should contact TimelineReader for application/attempt/container report: Resolved
• Support HBase 2.0.0-beta1 as ATSv2 backend: Resolved
• Improve error message for invalid value in fields attribute: Resolved
• Give an informative message when incorrect format is used in ATSv2 filter attributes: Resolved
• ATS v2 returns invalid YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS from NM: Resolved
• Incorrect HTTP status code returned by ATSv2 for non-whitelisted users: Resolved
• HTTPS Ats v2 api call fails with "bad HTTP parsed": Resolved
• Race condition when container events are published for KILLED applications: Resolved
• Adding JMX Metrics for Timeline Collector and Reader: Resolved
• Provide Java client for fetching Yarn specific entities from TimelineReader: Resolved
• Document behavior of YARN-5742: Resolved
• ApplicationCLI should have option to take clusterId: Resolved
• [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls: Resolved
• HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage down: Resolved
• Publish container killed, paused and resumed events to ATSv2.: Resolved
• ATS v2 Generic TimelineStorageMonitor: Resolved
• Username splits won't help timelineservice.app_flow table: Resolved
• Update document for ATS HBase Custom tablenames (-entityTableName): Resolved
• HBaseTimelineSchemaCreator skipExistingTable should honor custom hbase tablenames: Resolved
• FlowActivity and FlowRun table prefix is wrong: Resolved
• ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics: Resolved
• RM Crashes with NPE at TimelineServiceV2Publisher.putEntity: Resolved