Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5355

YARN Timeline Service v.2: alpha 2

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0-beta1
    • timelineserver
    • None
    • Reviewed
    • Hide
      We are releasing the alpha2 version of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation.

      YARN Timeline Service v.2 alpha1 was introduced in 3.0.0-alpha1 via YARN-2928.

      YARN Timeline Service v.2 alpha2 is now being provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. Security is provided via Kerberos Authentication and delegation tokens. There is also a simple read level authorization provided via whitelists.

      Some of the notable improvements since alpha-1 are:
      - Security via Kerberos Authentication and delegation tokens
      - Read side simple authorization via whitelist
      - Client configurable entity sort ordering
      - New REST APIs for apps, app attempts, containers, fetching metrics by timerange, pagination, sub-app entities
      - Support for storing sub-application entities (entities that exist outside the scope of an application)
      - Configurable TTLs (time-to-live) for tables, configurable table prefixes, configurable hbase cluster
      - Flow level aggregations done as dynamic (table level) coprocessors
      - Uses latest stable HBase release 1.2.6

      More details are available in the [YARN Timeline Service v.2](./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html) documentation.
      Show
      We are releasing the alpha2 version of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation. YARN Timeline Service v.2 alpha1 was introduced in 3.0.0-alpha1 via YARN-2928 . YARN Timeline Service v.2 alpha2 is now being provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. Security is provided via Kerberos Authentication and delegation tokens. There is also a simple read level authorization provided via whitelists. Some of the notable improvements since alpha-1 are: - Security via Kerberos Authentication and delegation tokens - Read side simple authorization via whitelist - Client configurable entity sort ordering - New REST APIs for apps, app attempts, containers, fetching metrics by timerange, pagination, sub-app entities - Support for storing sub-application entities (entities that exist outside the scope of an application) - Configurable TTLs (time-to-live) for tables, configurable table prefixes, configurable hbase cluster - Flow level aggregations done as dynamic (table level) coprocessors - Uses latest stable HBase release 1.2.6 More details are available in the [YARN Timeline Service v.2](./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html) documentation.

    Description

      This is an umbrella JIRA for the alpha 2 milestone for YARN Timeline Service v.2.

      This is developed on feature branches: YARN-5355 for the trunk-based development and YARN-5355-branch-2 to maintain backports to branch-2. Any subtask work on this JIRA will be committed to those 2 branches.

      Attachments

        1. YARN-5355-branch-2.01.patch
          2.36 MB
          Vrushali C
        2. YARN-5355.03.patch
          1.21 MB
          Varun Saxena
        3. YARN-5355.02.patch
          1.21 MB
          Varun Saxena
        4. YARN-5355.01.patch
          1.21 MB
          Varun Saxena
        5. Timeline Service v2_ Ideas for Next Steps.pdf
          132 kB
          Sangjin Lee
        6. Documentation - The YARN Timeline Service v2.pdf
          297 kB
          Varun Saxena

        Issue Links

        1.
        [Data Serving] Implement new flow-based ATS queries in the new ATS design Sub-task Resolved Vrushali C Actions
        2.
        [Security] Review and implement authentication in ATS v.2 Sub-task Resolved Varun Saxena Actions
        3.
        Recover collector list when RM fails over Sub-task Resolved Li Lu Actions
        4.
        Fix timeline client retry Sub-task Resolved Haibo Chen Actions
        5.
        Allow configurable prefix for hbase table names (like prod, exp, test etc) Sub-task Resolved Vrushali C Actions
        6.
        Entity created time should be part of the row key of entity table Sub-task Resolved Zhijie Shen Actions
        7.
        Create POC timeline web UI with new YARN web UI framework Sub-task Resolved Li Lu Actions
        8.
        [Storage implementation] Support getEntities with only Application id but no userId Sub-task Resolved Li Lu Actions
        9.
        Support fetching metrics by time range Sub-task Resolved Varun Saxena Actions
        10.
        Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2 Sub-task Resolved Naganarasimha G R Actions
        11.
        Split TestHBaseTimelineStorage into multiple test classes Sub-task Resolved Varun Saxena Actions
        12.
        Refactor the coprocessor code & other definition classes into independent packages Sub-task Resolved Haibo Chen Actions
        13.
        YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state Sub-task Resolved Vrushali C Actions
        14.
        Refactor #isApplicationEntity and #getApplicationEvent from HBaseTimelineWriterImpl Sub-task Resolved Vrushali C Actions
        15.
        Make HBase configuration for the timeline service configurable Sub-task Resolved Joep Rottinghuis Actions
        16.
        Accommodate app-id->cluster mapping Sub-task Resolved Sangjin Lee Actions
        17.
        [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST Sub-task Resolved Rohith Sharma K S Actions
        18.
        [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters Sub-task Resolved Rohith Sharma K S Actions
        19.
        Persist collector discovery information to support RM HA Sub-task Resolved Li Lu Actions
        20.
        Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery Sub-task Resolved Li Lu Actions
        21.
        [ATSv2 Security] Collector side changes for loading auth filters and principals Sub-task Resolved Varun Saxena Actions
        22.
        [ATSv2 Security] Client side changes for authentication Sub-task Resolved Varun Saxena Actions
        23.
        Retrospect yarn entity fields which are publishing in events info fields. Sub-task Resolved Rohith Sharma K S Actions
        24.
        introduce entity prefix for return and sort order Sub-task Resolved Rohith Sharma K S Actions
        25.
        Run PerNodeTimelineCollectorsAuxService in a system container Sub-task Resolved Haibo Chen Actions
        26.
        Provide timeline reader API to list available timeline entity types for one application Sub-task Resolved Li Lu Actions
        27.
        adopt the id prefix for YARN, MR, and DS entities Sub-task Resolved Varun Saxena Actions
        28.
        Report only local collectors on node upon resync with RM after RM fails over Sub-task Resolved Haibo Chen Actions
        29.
        Update hbase version to 1.2 (removes phoenix dependencies) Sub-task Resolved Vrushali C Actions
        30.
        Update documentation for single node hbase deploy Sub-task Resolved Vrushali C Actions
        31.
        Support fromid(offset) filter for /flows API Sub-task Resolved Rohith Sharma K S Actions
        32.
        Eliminate timelineServiceV2 boolean flag in TimelineClientImpl Sub-task Resolved Unassigned Actions
        33.
        Support fromId for flowRuns and flow/flowRun apps REST API's Sub-task Resolved Rohith Sharma K S Actions
        34.
        CORS support in timeline v2 Sub-task Resolved Rohith Sharma K S Actions
        35.
        FlowRunEntity does not deserialize long values correctly Sub-task Resolved Rohith Sharma K S Actions
        36.
        Update the coprocessor to be a dynamically loaded one Sub-task Resolved Vrushali C Actions
        37.
        [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM Sub-task Resolved Varun Saxena Actions
        38.
        [ATSv2 Security] Renew delegation token for app automatically if an app collector is active Sub-task Resolved Varun Saxena Actions
        39.
        [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active Sub-task Resolved Varun Saxena Actions
        40.
        Add Builder methods for TimelineEntityFilters Sub-task Resolved Haibo Chen Actions
        41.
        TimelineReaderServer should wait to join with HttpServer2 Sub-task Resolved Sangjin Lee Actions
        42.
        Documentation changes for TimelineV2Client Sub-task Resolved Naganarasimha G R Actions
        43.
        Move UID constant to TimelineReaderUtils Sub-task Resolved Rohith Sharma K S Actions
        44.
        FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops timestamp Sub-task Resolved Haibo Chen Actions
        45.
        Add FROM_ID info key for timeline entities in reader response. Sub-task Resolved Rohith Sharma K S Actions
        46.
        Provide help information and documentation for TimelineSchemaCreator Sub-task Resolved Haibo Chen Actions
        47.
        timeline service schema creator fails if executed from a remote machine Sub-task Resolved Sangjin Lee Actions
        48.
        Rolling upgrade/config change is broken on timeline v2. Sub-task Resolved Vrushali C Actions
        49.
        Make TimelineV2Client's drain timeout after stop configurable Sub-task Resolved Haibo Chen Actions
        50.
        Implement putEntitiesAsync API in TimelineCollector Sub-task Resolved Haibo Chen Actions
        51.
        App level aggregation should not consider metric values reported in the previous aggregation cycle Sub-task Resolved Varun Saxena Actions
        52.
        Exceptions caused by synchronous putEntities requests can be swallowed Sub-task Resolved Haibo Chen Actions
        53.
        NMTimelinePublisher#serviceStop does not stop timeline clients Sub-task Resolved Haibo Chen Actions
        54.
        ATSv2 HBase related tests fail due to guava version upgrade Sub-task Resolved Haibo Chen Actions
        55.
        [ATSv2] Can't retrieve more than 1000 versions of metrics in time series Sub-task Resolved Vrushali C Actions
        56.
        TimelineCollector is not stopped when an app finishes in RM Sub-task Resolved Varun Saxena Actions
        57.
        Enhance the timelinewriter.flush() race condition fix Sub-task Resolved Haibo Chen Actions
        58.
        Store application flow context in NM state store for work-preserving restart Sub-task Resolved Rohith Sharma K S Actions
        59.
        Findbugs warning in YARN-5355 branch Sub-task Resolved Vrushali C Actions
        60.
        Update exception message during timeline collector aux service initialization Sub-task Resolved Vrushali C Actions
        61.
        ConcurrentModificationException in TimelineCollectorManager while stopping RM Sub-task Resolved Haibo Chen Actions
        62.
        Allow metric TTL for Application table to be specified through cmd Sub-task Resolved Haibo Chen Actions
        63.
        [ATSv2 Security] Timeline reader side changes for loading auth filters and principals Sub-task Resolved Varun Saxena Actions
        64.
        The column family indicated in AppToFlowTable class javadoc is wrong Sub-task Resolved Haibo Chen Actions
        65.
        Remove columnFor() methods of Columns in HBaseTimeline backend Sub-task Resolved Haibo Chen Actions
        66.
        [ATSv2] Update HBase version to 1.2.6 Sub-task Resolved Vrushali C Actions
        67.
        Add table for storing sub-application entities Sub-task Resolved Vrushali C Actions
        68.
        Ensure sub-application user is extracted & sent to timeline service Sub-task Resolved Rohith Sharma K S Actions
        69.
        NPE in RM while setting collectors map in NodeHeartbeatResponse Sub-task Resolved Vrushali C Actions
        70.
        Restrict read access to timelineservice v2 data Sub-task Resolved Vrushali C Actions
        71.
        [ATSv2] NPE in RM while starting timeline collector on recovery after explicit failover Sub-task Resolved Varun Saxena Actions
        72.
        Ensure that supplemented timestamp is stored only for flow run metrics Sub-task Resolved Varun Saxena Actions
        73.
        Reader API for sub application entities Sub-task Resolved Rohith Sharma K S Actions
        74.
        Supplement timestamp for min start/max end time columns in flow run table to avoid overwrite Sub-task Resolved Vrushali C Actions
        75.
        Refactor AppLevelTimelineCollector such that RM does not have aggregator threads created Sub-task Resolved Vrushali C Actions
        76.
        Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat Sub-task Resolved Haibo Chen Actions
        77.
        [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol Sub-task Resolved Varun Saxena Actions
        78.
        Backport HADOOP-14386 to YARN-5355 branch Sub-task Resolved Vrushali C Actions
        79.
        [ATSv2] NPE while starting hbase co-processor when HBase authorization is enabled. Sub-task Resolved Rohith Sharma K S Actions
        80.
        [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled Sub-task Resolved Varun Saxena Actions
        81.
        Nodemanager NPE running jobs with security off Sub-task Resolved Varun Saxena Actions
        82.
        Documentation updates for TimelineService v2 Sub-task Resolved Rohith Sharma K S Actions
        83.
        Adding a NoOp timeline writer and reader plugin classes for ATSv2 Sub-task Resolved Prabha Manepalli Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vrushalic Vrushali C
            sjlee0 Sangjin Lee
            Votes:
            0 Vote for this issue
            Watchers:
            23 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment