Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5355

YARN Timeline Service v.2: alpha 2

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: timelineserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      We are releasing the alpha2 version of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation.

      YARN Timeline Service v.2 alpha1 was introduced in 3.0.0-alpha1 via YARN-2928.

      YARN Timeline Service v.2 alpha2 is now being provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. Security is provided via Kerberos Authentication and delegation tokens. There is also a simple read level authorization provided via whitelists.

      Some of the notable improvements since alpha-1 are:
      - Security via Kerberos Authentication and delegation tokens
      - Read side simple authorization via whitelist
      - Client configurable entity sort ordering
      - New REST APIs for apps, app attempts, containers, fetching metrics by timerange, pagination, sub-app entities
      - Support for storing sub-application entities (entities that exist outside the scope of an application)
      - Configurable TTLs (time-to-live) for tables, configurable table prefixes, configurable hbase cluster
      - Flow level aggregations done as dynamic (table level) coprocessors
      - Uses latest stable HBase release 1.2.6

      More details are available in the [YARN Timeline Service v.2](./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html) documentation.
      Show
      We are releasing the alpha2 version of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation. YARN Timeline Service v.2 alpha1 was introduced in 3.0.0-alpha1 via YARN-2928 . YARN Timeline Service v.2 alpha2 is now being provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. Security is provided via Kerberos Authentication and delegation tokens. There is also a simple read level authorization provided via whitelists. Some of the notable improvements since alpha-1 are: - Security via Kerberos Authentication and delegation tokens - Read side simple authorization via whitelist - Client configurable entity sort ordering - New REST APIs for apps, app attempts, containers, fetching metrics by timerange, pagination, sub-app entities - Support for storing sub-application entities (entities that exist outside the scope of an application) - Configurable TTLs (time-to-live) for tables, configurable table prefixes, configurable hbase cluster - Flow level aggregations done as dynamic (table level) coprocessors - Uses latest stable HBase release 1.2.6 More details are available in the [YARN Timeline Service v.2](./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html) documentation.

      Description

      This is an umbrella JIRA for the alpha 2 milestone for YARN Timeline Service v.2.

      This is developed on feature branches: YARN-5355 for the trunk-based development and YARN-5355-branch-2 to maintain backports to branch-2. Any subtask work on this JIRA will be committed to those 2 branches.

        Attachments

        1. YARN-5355-branch-2.01.patch
          2.36 MB
          Vrushali C
        2. YARN-5355.03.patch
          1.21 MB
          Varun Saxena
        3. YARN-5355.02.patch
          1.21 MB
          Varun Saxena
        4. YARN-5355.01.patch
          1.21 MB
          Varun Saxena
        5. Timeline Service v2_ Ideas for Next Steps.pdf
          132 kB
          Sangjin Lee
        6. Documentation - The YARN Timeline Service v2.pdf
          297 kB
          Varun Saxena

          Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. YARN-5814 Add druid as storage backend in YARN Timeline Service

          • Major - Major loss of function.
          • Open
          relates to

          New Feature - A new feature of the product, which has yet to be developed. YARN-2928 YARN Timeline Service v.2: alpha 1

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-6732 mapreduce tasks for YARN Timeline Service v.2: alpha 2

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. YARN-7055 YARN Timeline Service v.2: beta 1 / GA

          • Major - Major loss of function.
          • Resolved
          1.
          		 [Data Serving] Implement new flow-based ATS queries in the new ATS design Sub-task Resolved Vrushali C
          2.
          		 [Security] Review and implement authentication in ATS v.2 Sub-task Resolved Varun Saxena
          3.
          		 Recover collector list when RM fails over Sub-task Resolved Li Lu
          4.
          		 Fix timeline client retry Sub-task Resolved Haibo Chen
          5.
          		 Allow configurable prefix for hbase table names (like prod, exp, test etc) Sub-task Resolved Vrushali C
          6.
          		 Entity created time should be part of the row key of entity table Sub-task Resolved Zhijie Shen
          7.
          		 Create POC timeline web UI with new YARN web UI framework Sub-task Resolved Li Lu
          8.
          		 [Storage implementation] Support getEntities with only Application id but no userId Sub-task Resolved Li Lu
          9.
          		 Support fetching metrics by time range Sub-task Resolved Varun Saxena
          10.
          		 Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2 Sub-task Resolved Naganarasimha G R
          11.
          		 Split TestHBaseTimelineStorage into multiple test classes Sub-task Resolved Varun Saxena
          12.
          		 Refactor the coprocessor code & other definition classes into independent packages Sub-task Resolved Haibo Chen
          13.
          		 YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state Sub-task Resolved Vrushali C
          14.
          		 Refactor #isApplicationEntity and #getApplicationEvent from HBaseTimelineWriterImpl Sub-task Resolved Vrushali C
          15.
          		 Make HBase configuration for the timeline service configurable Sub-task Resolved Joep Rottinghuis
          16.
          		 Accommodate app-id->cluster mapping Sub-task Resolved Sangjin Lee
          17.
          		 [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST Sub-task Resolved Rohith Sharma K S
          18.
          		 [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters Sub-task Resolved Rohith Sharma K S
          19.
          		 Persist collector discovery information to support RM HA Sub-task Resolved Li Lu
          20.
          		 Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery Sub-task Resolved Li Lu
          21.
          		 [ATSv2 Security] Collector side changes for loading auth filters and principals Sub-task Resolved Varun Saxena
          22.
          		 [ATSv2 Security] Client side changes for authentication Sub-task Resolved Varun Saxena
          23.
          		 Retrospect yarn entity fields which are publishing in events info fields. Sub-task Resolved Rohith Sharma K S
          24.
          		 introduce entity prefix for return and sort order Sub-task Resolved Rohith Sharma K S
          25.
          		 Run PerNodeTimelineCollectorsAuxService in a system container Sub-task Resolved Haibo Chen
          26.
          		 Provide timeline reader API to list available timeline entity types for one application Sub-task Resolved Li Lu
          27.
          		 adopt the id prefix for YARN, MR, and DS entities Sub-task Resolved Varun Saxena
          28.
          		 Report only local collectors on node upon resync with RM after RM fails over Sub-task Resolved Haibo Chen
          29.
          		 Update hbase version to 1.2 (removes phoenix dependencies) Sub-task Resolved Vrushali C
          30.
          		 Update documentation for single node hbase deploy Sub-task Resolved Vrushali C
          31.
          		 Support fromid(offset) filter for /flows API Sub-task Resolved Rohith Sharma K S
          32.
          		 Eliminate timelineServiceV2 boolean flag in TimelineClientImpl Sub-task Resolved Unassigned
          33.
          		 Support fromId for flowRuns and flow/flowRun apps REST API's Sub-task Resolved Rohith Sharma K S
          34.
          		 CORS support in timeline v2 Sub-task Resolved Rohith Sharma K S
          35.
          		 FlowRunEntity does not deserialize long values correctly Sub-task Resolved Rohith Sharma K S
          36.
          		 Update the coprocessor to be a dynamically loaded one Sub-task Resolved Vrushali C
          37.
          		 [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM Sub-task Resolved Varun Saxena
          38.
          		 [ATSv2 Security] Renew delegation token for app automatically if an app collector is active Sub-task Resolved Varun Saxena
          39.
          		 [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active Sub-task Resolved Varun Saxena
          40.
          		 Add Builder methods for TimelineEntityFilters Sub-task Resolved Haibo Chen
          41.
          		 TimelineReaderServer should wait to join with HttpServer2 Sub-task Resolved Sangjin Lee
          42.
          		 Documentation changes for TimelineV2Client Sub-task Resolved Naganarasimha G R
          43.
          		 Move UID constant to TimelineReaderUtils Sub-task Resolved Rohith Sharma K S
          44.
          		 FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops timestamp Sub-task Resolved Haibo Chen
          45.
          		 Add FROM_ID info key for timeline entities in reader response. Sub-task Resolved Rohith Sharma K S
          46.
          		 Provide help information and documentation for TimelineSchemaCreator Sub-task Resolved Haibo Chen
          47.
          		 timeline service schema creator fails if executed from a remote machine Sub-task Resolved Sangjin Lee
          48.
          		 Rolling upgrade/config change is broken on timeline v2. Sub-task Resolved Vrushali C
          49.
          		 Make TimelineV2Client's drain timeout after stop configurable Sub-task Resolved Haibo Chen
          50.
          		 Implement putEntitiesAsync API in TimelineCollector Sub-task Resolved Haibo Chen
          51.
          		 App level aggregation should not consider metric values reported in the previous aggregation cycle Sub-task Resolved Varun Saxena
          52.
          		 Exceptions caused by synchronous putEntities requests can be swallowed Sub-task Resolved Haibo Chen
          53.
          		 NMTimelinePublisher#serviceStop does not stop timeline clients Sub-task Resolved Haibo Chen
          54.
          		 ATSv2 HBase related tests fail due to guava version upgrade Sub-task Resolved Haibo Chen
          55.
          		 [ATSv2] Can't retrieve more than 1000 versions of metrics in time series Sub-task Resolved Vrushali C
          56.
          		 TimelineCollector is not stopped when an app finishes in RM Sub-task Resolved Varun Saxena
          57.
          		 Enhance the timelinewriter.flush() race condition fix Sub-task Resolved Haibo Chen
          58.
          		 Store application flow context in NM state store for work-preserving restart Sub-task Resolved Rohith Sharma K S
          59.
          		 Findbugs warning in YARN-5355 branch Sub-task Resolved Vrushali C
          60.
          		 Update exception message during timeline collector aux service initialization Sub-task Resolved Vrushali C
          61.
          		 ConcurrentModificationException in TimelineCollectorManager while stopping RM Sub-task Resolved Haibo Chen
          62.
          		 Allow metric TTL for Application table to be specified through cmd Sub-task Resolved Haibo Chen
          63.
          		 [ATSv2 Security] Timeline reader side changes for loading auth filters and principals Sub-task Resolved Varun Saxena
          64.
          		 The column family indicated in AppToFlowTable class javadoc is wrong Sub-task Resolved Haibo Chen
          65.
          		 Remove columnFor() methods of Columns in HBaseTimeline backend Sub-task Resolved Haibo Chen
          66.
          		 [ATSv2] Update HBase version to 1.2.6 Sub-task Resolved Vrushali C
          67.
          		 Add table for storing sub-application entities Sub-task Resolved Vrushali C
          68.
          		 Ensure sub-application user is extracted & sent to timeline service Sub-task Resolved Rohith Sharma K S
          69.
          		 NPE in RM while setting collectors map in NodeHeartbeatResponse Sub-task Resolved Vrushali C
          70.
          		 Restrict read access to timelineservice v2 data Sub-task Resolved Vrushali C
          71.
          		 [ATSv2] NPE in RM while starting timeline collector on recovery after explicit failover Sub-task Resolved Varun Saxena
          72.
          		 Ensure that supplemented timestamp is stored only for flow run metrics Sub-task Resolved Varun Saxena
          73.
          		 Reader API for sub application entities Sub-task Resolved Rohith Sharma K S
          74.
          		 Supplement timestamp for min start/max end time columns in flow run table to avoid overwrite Sub-task Resolved Vrushali C
          75.
          		 Refactor AppLevelTimelineCollector such that RM does not have aggregator threads created Sub-task Resolved Vrushali C
          76.
          		 Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat Sub-task Resolved Haibo Chen
          77.
          		 [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol Sub-task Resolved Varun Saxena
          78.
          		 Backport HADOOP-14386 to YARN-5355 branch Sub-task Resolved Vrushali C
          79.
          		 [ATSv2] NPE while starting hbase co-processor when HBase authorization is enabled. Sub-task Resolved Rohith Sharma K S
          80.
          		 [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled Sub-task Resolved Varun Saxena
          81.
          		 Nodemanager NPE running jobs with security off Sub-task Resolved Varun Saxena
          82.
          		 Documentation updates for TimelineService v2 Sub-task Resolved Rohith Sharma K S
          83.
          		 Adding a NoOp timeline writer and reader plugin classes for ATSv2 Sub-task Resolved Prabha Manepalli

            Activity

              People

              • Assignee:
                vrushalic Vrushali C
                Reporter:
                sjlee0 Sangjin Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: