[YARN-9802] YARN Timeline Service v2 (post GA features)

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Vrushali C
• Created: 29/Aug/19 17:18
• Updated: 23/Oct/19 16:15

Description

Epic jira to track new feature enhancements or issues after the next release of Timeline Service v2

Issue Links

• Parent Feature: MAPREDUCE-6943 Mapreduce tasks for YARN Timeline Service v.2: (post GA features)

Sub-Tasks

• Support ACLs in ATSv2: In Progress
• Support metric filters for flow runs: In Progress
• RM crashes due to runtime errors in TimelineServiceV2Publisher: Resolved
• Recover domain from backend for TimelineCollector.: In Progress
• [atsv2 read acls] Coprocessor for reader authorization check: In Progress
• HBaseTimelineSchemaCreator has to allow user to configure pre-splits: Patch Available
• AM get timeline service info from RM rather than Application specific configuration.: Patch Available
• GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table: Resolved
• Use hbase shaded jars: In Progress
• Use app user to publish entities from NM: Patch Available
• Support for listing flows with filter userid: Patch Available
• [Fault tolerance] Fault tolerant writer for timeline v2: In Progress
• Do ACLs check while retrieving entity-types per application: Resolved
• [atsv2 read acls] REST API to list domain/domains: Patch Available
• @Path("/apps/{appid}/appattempts") error message misleads: Patch Available
• Do ACLs check for flowrun entities: Open
• Enable timeline collector fault tolerance: Open
• [atsv2 read acls] Update timeline collector to include domain for entities: Open
• [atsv2 read acls] Update NM to include ACLs for system entities: Open
• [Security] users should not be able to create a generic TimelineEntity and associate arbitrary type: Open
• Do ACLs check for flow activity entities: Open
• [ATSv2 Security] Delegation token support for Timeline reader: Open
• Do ACLs check for generic entities: Open
• Support for new REST end point /clusterids: Open
• [atsv2 read acls] Enable hbase writer to store domain/ACLs info: Open
• [atsv2 read acls] Add a API TimelineV2Client#putDomain to publish domain information.: Open
• Allow YARN ATSv2 ACLs to be disabled: Open
• Timeline service v2 integration with Federation: Open
• Add a yarn application cost calculation framework in TimelineService v2: Open
• Do ACLs check for sub app entities: Open
• [ATSv2 Security] Use token for communication between NM and node collector: Open
• [ATSv2 Security] Consider using delegation token for publishing entities from NM: Open
• create a REST API that returns the clusters for a given app id: Open
• Do ACLs check for application entities: Open
• Support substring match in filters: Open
• Publish federation events to ATSv2.: Open
• Launch app collectors for unmanaged AMs': Open
• Coprocessor related code changes/cleanup pending HBASE-17273: Open
• Improve debugging on ATSv2 reader server: Open
• Final Status of applications shown as UNDEFINED in ATS app queries: Reopened
• Support data retention policy in YARN ATSv2: Open
• Writing more RM side app-level metrics: Open
• Support for collapse filter for flow entities with in the boundary of date range.: Open
• Review / Recommendations for hbase writer code: Open
• Issues with HBaseTimelineWriterImpl in single node hadoop & hbase cluster: Open
• refactor ColumnHelper read* methods: Open
• fix the order of timelinereader in yarn/yarn.cmd: Resolved
• Support for listing all applications i.e /apps: Open
• Documentation updates for branch2 Timeline Service V2: Open
• address HBase compatibility issues with trunk: Resolved
• Timeline entity uniqueness: Open
• Limit flow runs returned while querying flows: Open
• Support Multiple versions of the timeline service at the same time: Open
• Need to support Application History Server on ATSV2: Resolved
• Update timeline collector to store and add domain info to entities while writing: Open
• Scalability validation of RM writing app/attempt/container lifecycle events: Open
• Backward compatibility of reader with ATSv1: Resolved
• AMs are unable to publish events if NMCollectorService starts at dynamic port.: Open
• Compaction coprocessor enhancements: On/Off, whitelisting, blacklisting: Open
• Address race condition on TimelineWriter.flush() caused by buffer-sized flush: Open
• Bubble exceptions and errors all the way up the calls, including to clients.: Open
• Extend aggregation operation for new ATS design: Open
• timeline flow runs API createdtimestart/createdtimeend parameter does not work: Open
• For long running apps (> 2 days), populate flow activity table: Resolved
• Writing RM cluster-level metrics: Open
• Support UNIT for TimelineMetric: Open
• Provide generic log servlet for serving logs: Open
• Make async/sync writes to timeline service configurable: Open
• create the flow-version table: Open
• Consider adding a config setting to accept logical cluster name: Open
• Optimize and synchronize FS Reader and Writer Implementations: Open
• Timeline client won't be able to write when TimelineCollector is not up yet, or NM is down: Open
• Inline java doc with Hadoop formatter: Open
• Ensure the final values for metrics/events are emitted/stored at APP completion time: Open
• Support to alter table properties in HBaseTimelineSchemaCreator: Open
• Enhance hbase coprocessor aggregation operations:GLOBAL_MIN, LATEST_MIN etc and FINAL attributes: Open
• TestTimelineReaderWebServices times out on YARN-5355_branch2: Open
• Have a separate NM timeline publishing-interval: Open
• [ATSv2] Create HBase connection only if an app collector is publishing from NM: Resolved
• Change collector discovery to support collectors mapped to clients but not applications: Open
• offline collector: support timeline clients not associated with an application: Open
• Retrospect on defaults for created time while querying: Resolved
• [Atsv2] Support streaming reader API to fetch entities: Open
• Support long running apps handling multiple flows: Open
• NM hangs at serviceStop when ATSV2 Backend Hbase is Down: Resolved
• TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down.: Resolved
• TimelineService Hbase tests are failing: Open
• TimelineSchemaCreator breaks after protobuffer upgrade: Open
• TimelineV2Client may leak file descriptors creating ClientResponse objects.: Resolved
• Region in ats-hbase table 'prod.timelineservice.entity' failing to split: Open
• TimelineConnector does not destroy the jersey client: Resolved
• Prevent Fatal CancelledException in TimelineV2Client when stopping: Resolved
• TestRMHATimelineCollectors fails on hadoop trunk: Resolved
• FlowRunCoprocessor Scan Used Deprecated Method: In Progress