[YARN-7055] YARN Timeline Service v.2: beta 1

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: timelineclient, timelinereader, timelineserver
• Assignee: Vrushali C
• Reporter: Vrushali C
• Created: 19/Aug/17 05:22
• Updated: 19/Feb/19 07:18

Description

This is an umbrella JIRA for the beta 1 milestone for YARN Timeline Service v.2.

YARN-2928 was alpha1, YARN-5355 was alpha2.

Issue Links

• incorporates: YARN-7213 [Umbrella] Test and validate HBase-2.0.x with Atsv2
• is related to: YARN-5355 YARN Timeline Service v.2: alpha 2
• Parent Feature: MAPREDUCE-6943 Mapreduce tasks for YARN Timeline Service v.2: beta 1

Sub-Tasks

• Limit the flow name size & consider cleanup for hex chars: Resolved
• Ship single node HBase config option with single startup command: Open
• Explore supporting a simpler back-end implementation for ATS v2: Open
• Bubble exceptions and errors all the way up the calls, including to clients.: Open
• [Collector implementation] Implement timeline collector functionality to serve ATS readers' requests for live apps: Open
• [Collector wireup] Implement NM starting the standalone timeline collector daemon: Open
• [Collector wireup] Handle timeline collector failure scenarios: Open
• [Collector wireup] Work-preserving restarting of per-node timeline collector: Open
• [Data Serving] Clustering of ATS reader instances: Open
• [Source organization] Decide detailed package structures for timeline service v2 components: Open
• [Compatibility] Make TS next gen be compatible with the current TS: Open
• [Event producers] Change distributed shell to use TIMELINE_SERVICE_VERSION: Open
• provide RPC metrics via JMX for timeline collectors and readers: Resolved
• [Security] users should not be able to create a generic TimelineEntity and associate arbitrary type: Open
• CollectorNodemanagerProtocol's authorization doesn't work: Resolved
• AM get timeline service info from RM rather than Application specific configuration.: Patch Available
• Investigate the concurrency issue with the map of timeline collector: Open
• Timeline entity uniqueness: Open
• determine how to generate YARN container events: Open
• Enable application client to communicate with new timeline service: Open
• Consider concurrency situations for TimelineWriter: Open
• consider nulling member maps/sets of TimelineEntity: Open
• [Aggregation] Application/Flow/User/Queue Level Aggregations: Open
• [Aggregation] Flow and User level aggregation on Application States table: In Progress
• [Aggregation] Queue-level Aggregation on Application States table: Open
• Scalability validation of RM writing app/attempt/container lifecycle events: Open
• Backward compatibility of reader with ATSv1: Open
• TimelineReader Web UI Implementation: Resolved
• Optimize and synchronize FS Reader and Writer Implementations: Open
• [Fault tolerance] Fault tolerant writer for timeline v2: In Progress
• Writing more RM side app-level metrics: Open
• Writing RM cluster-level metrics: Open
• Support ACLs in ATSv2: In Progress
• create the flow-version table: Open
• offline collector: support timeline clients not associated with an application: Open
• For long running apps (> 2 days), populate flow activity table: Open
• refactor ColumnHelper read* methods: Open
• Ensure the final values for metrics/events are emitted/stored at APP completion time: Open
• Flow page for Web UI: Resolved
• fix the order of timelinereader in yarn/yarn.cmd: Open
• Need to support Application History Server on ATSV2: Open
• Support Multiple versions of the timeline service at the same time: Open
• Limit flow runs returned while querying flows: Open
• Retrospect on defaults for created time while querying: Open
• Compaction coprocessor enhancements: On/Off, whitelisting, blacklisting: Open
• Issues with HBaseTimelineWriterImpl in single node hadoop & hbase cluster: Open
• Enhance hbase coprocessor aggregation operations:GLOBAL_MIN, LATEST_MIN etc and FINAL attributes: Open
• Have a separate NM timeline publishing-interval: Open
• Support metric filters for flow runs: In Progress
• address HBase compatibility issues with trunk: Open
• some YARN container events have timestamp of -1: Resolved
• Review / Recommendations for hbase writer code: Open
• Timeline service v2 integration with Federation: Open
• Consider adding a config setting to accept logical cluster name: Open
• Support long running apps handling multiple flows: Open
• [Atsv2] Support streaming reader API to fetch entities: Open
• Change collector discovery to support collectors mapped to clients but not applications: Open
• Serve aggregated logs of historical apps from timeline service: Resolved
• Support UNIT for TimelineMetric: Open
• [ATSv2] Create HBase connection only if an app collector is publishing from NM: Open
• Coprocessor related code changes/cleanup pending HBASE-17273: Open
• Support for listing all applications i.e /apps: Open
• Inline java doc with Hadoop formatter: Open
• Support substring match in filters: Open
• create a REST API that returns the clusters for a given app id: Open
• Support for new REST end point /clusterids: Open
• [Storage implementation] Create HDFS backing storage implementation for ATS reads: Resolved
• Launch app collectors for unmanaged AMs': Open
• Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a consistent way: Resolved
• [ATSv2] : NPE for recovered finished applications in NM during upgrade: Resolved
• Make the aggregation interval in AppLevelTimelineCollector configurable: Resolved
• [ATSv2] Fix findbugs warnings: Resolved
• Timeline client won't be able to write when TimelineCollector is not up yet, or NM is down: Open
• Consider writing to both ats v1 & v2 from RM for smoother upgrades: Resolved
• Have a way to turn off container metrics from NMs: Resolved
• Address race condition on TimelineWriter.flush() caused by buffer-sized flush: Open
• Support for listing flows with filter userid: Patch Available
• Support for collapse filter for flow entities with in the boundary of date range.: Open
• Use app user to publish entities from NM: Patch Available
• [Storage implementation] Adding retry semantics to HDFS backing storage: Resolved
• Allow port range to be specified while starting NM Timeline collector manager.: Resolved
• start time key in NM leveldb store should be removed when container is removed: Resolved
• [ATSv2 Security] Delegation token support for Timeline reader: Open
• [ATSv2 Security] Consider using delegation token for publishing entities from NM: Open
• [ATSv2 Security] Use token for communication between NM and node collector: Open
• Documentation updates for branch2 Timeline Service V2: Open
• Extend aggregation operation for new ATS design: Open
• Fix findbugs warning in timeline service on branch-2: Resolved
• The error message in TimelineSchemaCreator is not enough to find out the error.: Resolved
• Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2): Resolved
• TestTimelineReaderWebServicesHBaseStorage fails with NoClassDefFoundError on TSv2 branch2: Resolved
• Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath: Resolved
• Add a yarn application cost calculation framework in TimelineService v2: Open
• Enable timeline collector fault tolerance: Open
• Backport ATSv2 to branch-2: Resolved
• [atsv2 read acls] Include domain table creation as part of schema creator: Resolved
• TestTimelineReaderWebServices times out on YARN-5355_branch2: Open
• Documentation changes post branch-2 merge: Resolved
• Update LICENSE.txt with respect to new YARN UI merge: Resolved
• [atsv2 read acls] REST API to list domain/domains: Patch Available
• [Atsv2] load hbase configuration from filesystem rather than URL: Resolved
• YARN_TIMELINESERVER_OPTS is not valid env variable for TimelineReaderServer: Resolved
• metricsTimeStart and metricsTimeEnd should be all lower case in the doc: Resolved
• [Atsv2] Update document for running v1 and v2 TS: Resolved
• FlowRunReader constructs min_start_time filter for both createdtimestart and createdtimeend.: Resolved
• [atsv2 read acls] Add a API TimelineV2Client#putDomain to publish domain information.: Open
• [atsv2 read acls] Add TimelineWriter#writeDomain: Resolved
• Do ACLs check for flow activity entities: Open
• Do ACLs check for flowrun entities: Open
• Do ACLs check for application entities: Open
• Do ACLs check for generic entities: Open
• Do ACLs check for sub app entities: Open
• Do ACLs check while retrieving entity-types per application: Patch Available
• [Atsv2] Retrospect storing entities into sub application table from client perspective: Resolved
• Add table for storing timeline domain information: Resolved
• Support data retention policy in YARN ATSv2: Open
• Improve debugging on ATSv2 reader server: Open
• ATSv2 REST API queries do not return results for uppercase application tags: Resolved
• Allow YARN ATSv2 ACLs to be disabled: Open
• [atsv2 read acls] REST API to get domain info by id: Resolved
• Recover domain from backend for TimelineCollector.: In Progress
• YarnClient should contact TimelineReader for application/attempt/container report: Resolved
• Support HBase 2.0.0-beta1 as ATSv2 backend: Resolved
• Update timeline collector to store and add domain info to entities while writing: Open
• Improve error message for invalid value in fields attribute: Resolved
• Give an informative message when incorrect format is used in ATSv2 filter attributes: Resolved
• Final Status of applications shown as UNDEFINED in ATS app queries: Reopened
• ATS v2 returns invalid YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS from NM: Resolved
• Incorrect HTTP status code returned by ATSv2 for non-whitelisted users: Resolved
• [atsv2 read acls] Update timeline collector to include domain for entities: Open
• HTTPS Ats v2 api call fails with "bad HTTP parsed": Resolved
• Race condition when container events are published for KILLED applications: Resolved
• Provide generic log servlet for serving logs: Open
• Adding JMX Metrics for Timeline Collector and Reader: Resolved
• [atsv2 read acls] Enable hbase writer to store domain/ACLs info: Open
• [atsv2 read acls] Update NM to include ACLs for system entities: Open
• [atsv2 read acls] Coprocessor for reader authorization check: In Progress
• timeline flow runs API createdtimestart/createdtimeend parameter does not work: Open
• Use hbase shaded jars: In Progress
• AMs are unable to publish events if NMCollectorService starts at dynamic port.: Open
• Provide Java client for fetching Yarn specific entities from TimelineReader: Resolved
• Document behavior of YARN-5742: Resolved
• ApplicationCLI should have option to take clusterId: Resolved
• [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls: Resolved
• HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage down: Resolved
• Publish container killed, paused and resumed events to ATSv2.: Resolved
• Publish federation events to ATSv2.: Open
• ATS v2 Generic TimelineStorageMonitor: Resolved
• Username splits won't help timelineservice.app_flow table: Resolved
• HBaseTimelineSchemaCreator has to allow user to configure pre-splits: Patch Available
• Update document for ATS HBase Custom tablenames (-entityTableName): Resolved
• HBaseTimelineSchemaCreator skipExistingTable should honor custom hbase tablenames: Resolved
• FlowActivity and FlowRun table prefix is wrong: Resolved
• GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table: Patch Available
• ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics: Resolved
• @Path("/apps/{appid}/appattempts") error message misleads: Patch Available
• RM Crashes with NPE at TimelineServiceV2Publisher.putEntity: Resolved
• Support to alter table properties in HBaseTimelineSchemaCreator: Open