Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: timelineserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      We are releasing the alpha2 version of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation.

      YARN Timeline Service v.2 alpha1 was introduced in 3.0.0-alpha1 via YARN-2928.

      YARN Timeline Service v.2 alpha2 is now being provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. Security is provided via Kerberos Authentication and delegation tokens. There is also a simple read level authorization provided via whitelists.

      Some of the notable improvements since alpha-1 are:
      - Security via Kerberos Authentication and delegation tokens
      - Read side simple authorization via whitelist
      - Client configurable entity sort ordering
      - New REST APIs for apps, app attempts, containers, fetching metrics by timerange, pagination, sub-app entities
      - Support for storing sub-application entities (entities that exist outside the scope of an application)
      - Configurable TTLs (time-to-live) for tables, configurable table prefixes, configurable hbase cluster
      - Flow level aggregations done as dynamic (table level) coprocessors
      - Uses latest stable HBase release 1.2.6

      More details are available in the [YARN Timeline Service v.2](./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html) documentation.
      Show
      We are releasing the alpha2 version of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation. YARN Timeline Service v.2 alpha1 was introduced in 3.0.0-alpha1 via YARN-2928 . YARN Timeline Service v.2 alpha2 is now being provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. Security is provided via Kerberos Authentication and delegation tokens. There is also a simple read level authorization provided via whitelists. Some of the notable improvements since alpha-1 are: - Security via Kerberos Authentication and delegation tokens - Read side simple authorization via whitelist - Client configurable entity sort ordering - New REST APIs for apps, app attempts, containers, fetching metrics by timerange, pagination, sub-app entities - Support for storing sub-application entities (entities that exist outside the scope of an application) - Configurable TTLs (time-to-live) for tables, configurable table prefixes, configurable hbase cluster - Flow level aggregations done as dynamic (table level) coprocessors - Uses latest stable HBase release 1.2.6 More details are available in the [YARN Timeline Service v.2](./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html) documentation.

      Description

      This is an umbrella JIRA for the alpha 2 milestone for YARN Timeline Service v.2.

      This is developed on feature branches: YARN-5355 for the trunk-based development and YARN-5355-branch-2 to maintain backports to branch-2. Any subtask work on this JIRA will be committed to those 2 branches.

        Attachments

        1. Documentation - The YARN Timeline Service v2.pdf
          297 kB
          Varun Saxena
        2. Timeline Service v2_ Ideas for Next Steps.pdf
          132 kB
          Sangjin Lee
        3. YARN-5355.01.patch
          1.21 MB
          Varun Saxena
        4. YARN-5355.02.patch
          1.21 MB
          Varun Saxena
        5. YARN-5355.03.patch
          1.21 MB
          Varun Saxena
        6. YARN-5355-branch-2.01.patch
          2.36 MB
          Vrushali C

          Issue Links

          1.
          [Data Serving] Implement new flow-based ATS queries in the new ATS design Sub-task Resolved Vrushali C
          2.
          [Security] Review and implement authentication in ATS v.2 Sub-task Resolved Varun Saxena
          3.
          Recover collector list when RM fails over Sub-task Resolved Li Lu
          4.
          Fix timeline client retry Sub-task Resolved Haibo Chen
          5.
          Allow configurable prefix for hbase table names (like prod, exp, test etc) Sub-task Resolved Vrushali C
          6.
          Entity created time should be part of the row key of entity table Sub-task Resolved Zhijie Shen
          7.
          Create POC timeline web UI with new YARN web UI framework Sub-task Resolved Li Lu
          8.
          [Storage implementation] Support getEntities with only Application id but no userId Sub-task Resolved Li Lu
          9.
          Support fetching metrics by time range Sub-task Resolved Varun Saxena
          10.
          Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2 Sub-task Resolved Naganarasimha G R
          11.
          Split TestHBaseTimelineStorage into multiple test classes Sub-task Resolved Varun Saxena
          12.
          Refactor the coprocessor code & other definition classes into independent packages Sub-task Resolved Haibo Chen
          13.
          YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state Sub-task Resolved Vrushali C
          14.
          Refactor #isApplicationEntity and #getApplicationEvent from HBaseTimelineWriterImpl Sub-task Resolved Vrushali C
          15.
          Make HBase configuration for the timeline service configurable Sub-task Resolved Joep Rottinghuis
          16.
          Accommodate app-id->cluster mapping Sub-task Resolved Sangjin Lee
          17.
          [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST Sub-task Resolved Rohith Sharma K S
          18.
          [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters Sub-task Resolved Rohith Sharma K S
          19.
          Persist collector discovery information to support RM HA Sub-task Resolved Li Lu
          20.
          Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery Sub-task Resolved Li Lu
          21.
          [ATSv2 Security] Collector side changes for loading auth filters and principals Sub-task Resolved Varun Saxena
          22.
          [ATSv2 Security] Client side changes for authentication Sub-task Resolved Varun Saxena
          23.
          Retrospect yarn entity fields which are publishing in events info fields. Sub-task Resolved Rohith Sharma K S
          24.
          introduce entity prefix for return and sort order Sub-task Resolved Rohith Sharma K S
          25.
          Run PerNodeTimelineCollectorsAuxService in a system container Sub-task Resolved Haibo Chen
          26.
          Provide timeline reader API to list available timeline entity types for one application Sub-task Resolved Li Lu
          27.
          adopt the id prefix for YARN, MR, and DS entities Sub-task Resolved Varun Saxena
          28.
          Report only local collectors on node upon resync with RM after RM fails over Sub-task Resolved Haibo Chen
          29.
          Update hbase version to 1.2 (removes phoenix dependencies) Sub-task Resolved Vrushali C
          30.
          Update documentation for single node hbase deploy Sub-task Resolved Vrushali C
          31.
          Support fromid(offset) filter for /flows API Sub-task Resolved Rohith Sharma K S
          32.
          Eliminate timelineServiceV2 boolean flag in TimelineClientImpl Sub-task Resolved Unassigned
          33.
          Support fromId for flowRuns and flow/flowRun apps REST API's Sub-task Resolved Rohith Sharma K S
          34.
          CORS support in timeline v2 Sub-task Resolved Rohith Sharma K S
          35.
          FlowRunEntity does not deserialize long values correctly Sub-task Resolved Rohith Sharma K S
          36.
          Update the coprocessor to be a dynamically loaded one Sub-task Resolved Vrushali C
          37.
          [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM Sub-task Resolved Varun Saxena
          38.
          [ATSv2 Security] Renew delegation token for app automatically if an app collector is active Sub-task Resolved Varun Saxena
          39.
          [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active Sub-task Resolved Varun Saxena
          40.
          Add Builder methods for TimelineEntityFilters Sub-task Resolved Haibo Chen
          41.
          TimelineReaderServer should wait to join with HttpServer2 Sub-task Resolved Sangjin Lee
          42.
          Documentation changes for TimelineV2Client Sub-task Resolved Naganarasimha G R
          43.
          Move UID constant to TimelineReaderUtils Sub-task Resolved Rohith Sharma K S
          44.
          FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops timestamp Sub-task Resolved Haibo Chen
          45.
          Add FROM_ID info key for timeline entities in reader response. Sub-task Resolved Rohith Sharma K S
          46.
          Provide help information and documentation for TimelineSchemaCreator Sub-task Resolved Haibo Chen
          47.
          timeline service schema creator fails if executed from a remote machine Sub-task Resolved Sangjin Lee
          48.
          Rolling upgrade/config change is broken on timeline v2. Sub-task Resolved Vrushali C
          49.
          Make TimelineV2Client's drain timeout after stop configurable Sub-task Resolved Haibo Chen
          50.
          Implement putEntitiesAsync API in TimelineCollector Sub-task Resolved Haibo Chen
          51.
          App level aggregation should not consider metric values reported in the previous aggregation cycle Sub-task Resolved Varun Saxena
          52.
          Exceptions caused by synchronous putEntities requests can be swallowed Sub-task Resolved Haibo Chen
          53.
          NMTimelinePublisher#serviceStop does not stop timeline clients Sub-task Resolved Haibo Chen
          54.
          ATSv2 HBase related tests fail due to guava version upgrade Sub-task Resolved Haibo Chen
          55.
          [ATSv2] Can't retrieve more than 1000 versions of metrics in time series Sub-task Resolved Vrushali C
          56.
          TimelineCollector is not stopped when an app finishes in RM Sub-task Resolved Varun Saxena
          57.
          Enhance the timelinewriter.flush() race condition fix Sub-task Resolved Haibo Chen
          58.
          Store application flow context in NM state store for work-preserving restart Sub-task Resolved Rohith Sharma K S
          59.
          Findbugs warning in YARN-5355 branch Sub-task Resolved Vrushali C
          60.
          Update exception message during timeline collector aux service initialization Sub-task Resolved Vrushali C
          61.
          ConcurrentModificationException in TimelineCollectorManager while stopping RM Sub-task Resolved Haibo Chen
          62.
          Allow metric TTL for Application table to be specified through cmd Sub-task Resolved Haibo Chen
          63.
          [ATSv2 Security] Timeline reader side changes for loading auth filters and principals Sub-task Resolved Varun Saxena
          64.
          The column family indicated in AppToFlowTable class javadoc is wrong Sub-task Resolved Haibo Chen
          65.
          Remove columnFor() methods of Columns in HBaseTimeline backend Sub-task Resolved Haibo Chen
          66.
          [ATSv2] Update HBase version to 1.2.6 Sub-task Resolved Vrushali C
          67.
          Add table for storing sub-application entities Sub-task Resolved Vrushali C
          68.
          Ensure sub-application user is extracted & sent to timeline service Sub-task Resolved Rohith Sharma K S
          69.
          NPE in RM while setting collectors map in NodeHeartbeatResponse Sub-task Resolved Vrushali C
          70.
          Restrict read access to timelineservice v2 data Sub-task Resolved Vrushali C
          71.
          [ATSv2] NPE in RM while starting timeline collector on recovery after explicit failover Sub-task Resolved Varun Saxena
          72.
          Ensure that supplemented timestamp is stored only for flow run metrics Sub-task Resolved Varun Saxena
          73.
          Reader API for sub application entities Sub-task Resolved Rohith Sharma K S
          74.
          Supplement timestamp for min start/max end time columns in flow run table to avoid overwrite Sub-task Resolved Vrushali C
          75.
          Refactor AppLevelTimelineCollector such that RM does not have aggregator threads created Sub-task Resolved Vrushali C
          76.
          Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat Sub-task Resolved Haibo Chen
          77.
          [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol Sub-task Resolved Varun Saxena
          78.
          Backport HADOOP-14386 to YARN-5355 branch Sub-task Resolved Vrushali C
          79.
          [ATSv2] NPE while starting hbase co-processor when HBase authorization is enabled. Sub-task Resolved Rohith Sharma K S
          80.
          [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled Sub-task Resolved Varun Saxena
          81.
          Nodemanager NPE running jobs with security off Sub-task Resolved Varun Saxena
          82.
          Documentation updates for TimelineService v2 Sub-task Resolved Rohith Sharma K S

            Activity

              People

              • Assignee:
                vrushalic Vrushali C
                Reporter:
                sjlee0 Sangjin Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                21 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: