Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5355

YARN Timeline Service v.2: alpha 2

          Details

          • Type: New Feature
          • Status: Resolved
          • Priority: Critical
          • Resolution: Fixed
          • Affects Version/s: None
          • Fix Version/s: 2.9.0, 3.0.0-beta1
          • Component/s: timelineserver
          • Labels:
            None
          • Hadoop Flags:
            Reviewed
          • Release Note:
            Hide
            We are releasing the alpha2 version of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation.

            YARN Timeline Service v.2 alpha1 was introduced in 3.0.0-alpha1 via YARN-2928.

            YARN Timeline Service v.2 alpha2 is now being provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. Security is provided via Kerberos Authentication and delegation tokens. There is also a simple read level authorization provided via whitelists.

            Some of the notable improvements since alpha-1 are:
            - Security via Kerberos Authentication and delegation tokens
            - Read side simple authorization via whitelist
            - Client configurable entity sort ordering
            - New REST APIs for apps, app attempts, containers, fetching metrics by timerange, pagination, sub-app entities
            - Support for storing sub-application entities (entities that exist outside the scope of an application)
            - Configurable TTLs (time-to-live) for tables, configurable table prefixes, configurable hbase cluster
            - Flow level aggregations done as dynamic (table level) coprocessors
            - Uses latest stable HBase release 1.2.6

            More details are available in the [YARN Timeline Service v.2](./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html) documentation.
            Show
            We are releasing the alpha2 version of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation. YARN Timeline Service v.2 alpha1 was introduced in 3.0.0-alpha1 via YARN-2928 . YARN Timeline Service v.2 alpha2 is now being provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. Security is provided via Kerberos Authentication and delegation tokens. There is also a simple read level authorization provided via whitelists. Some of the notable improvements since alpha-1 are: - Security via Kerberos Authentication and delegation tokens - Read side simple authorization via whitelist - Client configurable entity sort ordering - New REST APIs for apps, app attempts, containers, fetching metrics by timerange, pagination, sub-app entities - Support for storing sub-application entities (entities that exist outside the scope of an application) - Configurable TTLs (time-to-live) for tables, configurable table prefixes, configurable hbase cluster - Flow level aggregations done as dynamic (table level) coprocessors - Uses latest stable HBase release 1.2.6 More details are available in the [YARN Timeline Service v.2](./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html) documentation.

            Description

            This is an umbrella JIRA for the alpha 2 milestone for YARN Timeline Service v.2.

            This is developed on feature branches: YARN-5355 for the trunk-based development and YARN-5355-branch-2 to maintain backports to branch-2. Any subtask work on this JIRA will be committed to those 2 branches.

              Attachments

              1. Documentation - The YARN Timeline Service v2.pdf
                297 kB
                Varun Saxena
              2. Timeline Service v2_ Ideas for Next Steps.pdf
                132 kB
                Sangjin Lee
              3. YARN-5355.01.patch
                1.21 MB
                Varun Saxena
              4. YARN-5355.02.patch
                1.21 MB
                Varun Saxena
              5. YARN-5355.03.patch
                1.21 MB
                Varun Saxena
              6. YARN-5355-branch-2.01.patch
                2.36 MB
                Vrushali C

                Issue Links

                is related to

                New Feature - A new feature of the product, which has yet to be developed. YARN-5814 Add druid as storage backend in YARN Timeline Service

                • Major - Major loss of function.
                • Open
                relates to

                New Feature - A new feature of the product, which has yet to be developed. YARN-7055 YARN Timeline Service v.2: beta 1

                • Major - Major loss of function.
                • Open

                New Feature - A new feature of the product, which has yet to be developed. YARN-2928 YARN Timeline Service v.2: alpha 1

                • Critical - Crashes, loss of data, severe memory leak.
                • Resolved

                New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-6732 mapreduce tasks for YARN Timeline Service v.2: alpha 2

                • Major - Major loss of function.
                • Resolved
                1.
                		 [Data Serving] Implement new flow-based ATS queries in the new ATS design Sub-task Resolved Vrushali C
                2.
                		 [Security] Review and implement authentication in ATS v.2 Sub-task Resolved Varun Saxena
                3.
                		 Recover collector list when RM fails over Sub-task Resolved Li Lu
                4.
                		 Fix timeline client retry Sub-task Resolved Haibo Chen
                5.
                		 Allow configurable prefix for hbase table names (like prod, exp, test etc) Sub-task Resolved Vrushali C
                6.
                		 Entity created time should be part of the row key of entity table Sub-task Resolved Zhijie Shen
                7.
                		 Create POC timeline web UI with new YARN web UI framework Sub-task Resolved Li Lu
                8.
                		 [Storage implementation] Support getEntities with only Application id but no userId Sub-task Resolved Li Lu
                9.
                		 Support fetching metrics by time range Sub-task Resolved Varun Saxena
                10.
                		 Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2 Sub-task Resolved Naganarasimha G R
                11.
                		 Split TestHBaseTimelineStorage into multiple test classes Sub-task Resolved Varun Saxena
                12.
                		 Refactor the coprocessor code & other definition classes into independent packages Sub-task Resolved Haibo Chen
                13.
                		 YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state Sub-task Resolved Vrushali C
                14.
                		 Refactor #isApplicationEntity and #getApplicationEvent from HBaseTimelineWriterImpl Sub-task Resolved Vrushali C
                15.
                		 Make HBase configuration for the timeline service configurable Sub-task Resolved Joep Rottinghuis
                16.
                		 Accommodate app-id->cluster mapping Sub-task Resolved Sangjin Lee
                17.
                		 [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST Sub-task Resolved Rohith Sharma K S
                18.
                		 [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters Sub-task Resolved Rohith Sharma K S
                19.
                		 Persist collector discovery information to support RM HA Sub-task Resolved Li Lu
                20.
                		 Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery Sub-task Resolved Li Lu
                21.
                		 [ATSv2 Security] Collector side changes for loading auth filters and principals Sub-task Resolved Varun Saxena
                22.
                		 [ATSv2 Security] Client side changes for authentication Sub-task Resolved Varun Saxena
                23.
                		 Retrospect yarn entity fields which are publishing in events info fields. Sub-task Resolved Rohith Sharma K S
                24.
                		 introduce entity prefix for return and sort order Sub-task Resolved Rohith Sharma K S
                25.
                		 Run PerNodeTimelineCollectorsAuxService in a system container Sub-task Resolved Haibo Chen
                26.
                		 Provide timeline reader API to list available timeline entity types for one application Sub-task Resolved Li Lu
                27.
                		 adopt the id prefix for YARN, MR, and DS entities Sub-task Resolved Varun Saxena
                28.
                		 Report only local collectors on node upon resync with RM after RM fails over Sub-task Resolved Haibo Chen
                29.
                		 Update hbase version to 1.2 (removes phoenix dependencies) Sub-task Resolved Vrushali C
                30.
                		 Update documentation for single node hbase deploy Sub-task Resolved Vrushali C
                31.
                		 Support fromid(offset) filter for /flows API Sub-task Resolved Rohith Sharma K S
                32.
                		 Eliminate timelineServiceV2 boolean flag in TimelineClientImpl Sub-task Resolved Unassigned
                33.
                		 Support fromId for flowRuns and flow/flowRun apps REST API's Sub-task Resolved Rohith Sharma K S
                34.
                		 CORS support in timeline v2 Sub-task Resolved Rohith Sharma K S
                35.
                		 FlowRunEntity does not deserialize long values correctly Sub-task Resolved Rohith Sharma K S
                36.
                		 Update the coprocessor to be a dynamically loaded one Sub-task Resolved Vrushali C
                37.
                		 [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM Sub-task Resolved Varun Saxena
                38.
                		 [ATSv2 Security] Renew delegation token for app automatically if an app collector is active Sub-task Resolved Varun Saxena
                39.
                		 [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active Sub-task Resolved Varun Saxena
                40.
                		 Add Builder methods for TimelineEntityFilters Sub-task Resolved Haibo Chen
                41.
                		 TimelineReaderServer should wait to join with HttpServer2 Sub-task Resolved Sangjin Lee
                42.
                		 Documentation changes for TimelineV2Client Sub-task Resolved Naganarasimha G R
                43.
                		 Move UID constant to TimelineReaderUtils Sub-task Resolved Rohith Sharma K S
                44.
                		 FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops timestamp Sub-task Resolved Haibo Chen
                45.
                		 Add FROM_ID info key for timeline entities in reader response. Sub-task Resolved Rohith Sharma K S
                46.
                		 Provide help information and documentation for TimelineSchemaCreator Sub-task Resolved Haibo Chen
                47.
                		 timeline service schema creator fails if executed from a remote machine Sub-task Resolved Sangjin Lee
                48.
                		 Rolling upgrade/config change is broken on timeline v2. Sub-task Resolved Vrushali C
                49.
                		 Make TimelineV2Client's drain timeout after stop configurable Sub-task Resolved Haibo Chen
                50.
                		 Implement putEntitiesAsync API in TimelineCollector Sub-task Resolved Haibo Chen
                51.
                		 App level aggregation should not consider metric values reported in the previous aggregation cycle Sub-task Resolved Varun Saxena
                52.
                		 Exceptions caused by synchronous putEntities requests can be swallowed Sub-task Resolved Haibo Chen
                53.
                		 NMTimelinePublisher#serviceStop does not stop timeline clients Sub-task Resolved Haibo Chen
                54.
                		 ATSv2 HBase related tests fail due to guava version upgrade Sub-task Resolved Haibo Chen
                55.
                		 [ATSv2] Can't retrieve more than 1000 versions of metrics in time series Sub-task Resolved Vrushali C
                56.
                		 TimelineCollector is not stopped when an app finishes in RM Sub-task Resolved Varun Saxena
                57.
                		 Enhance the timelinewriter.flush() race condition fix Sub-task Resolved Haibo Chen
                58.
                		 Store application flow context in NM state store for work-preserving restart Sub-task Resolved Rohith Sharma K S
                59.
                		 Findbugs warning in YARN-5355 branch Sub-task Resolved Vrushali C
                60.
                		 Update exception message during timeline collector aux service initialization Sub-task Resolved Vrushali C
                61.
                		 ConcurrentModificationException in TimelineCollectorManager while stopping RM Sub-task Resolved Haibo Chen
                62.
                		 Allow metric TTL for Application table to be specified through cmd Sub-task Resolved Haibo Chen
                63.
                		 [ATSv2 Security] Timeline reader side changes for loading auth filters and principals Sub-task Resolved Varun Saxena
                64.
                		 The column family indicated in AppToFlowTable class javadoc is wrong Sub-task Resolved Haibo Chen
                65.
                		 Remove columnFor() methods of Columns in HBaseTimeline backend Sub-task Resolved Haibo Chen
                66.
                		 [ATSv2] Update HBase version to 1.2.6 Sub-task Resolved Vrushali C
                67.
                		 Add table for storing sub-application entities Sub-task Resolved Vrushali C
                68.
                		 Ensure sub-application user is extracted & sent to timeline service Sub-task Resolved Rohith Sharma K S
                69.
                		 NPE in RM while setting collectors map in NodeHeartbeatResponse Sub-task Resolved Vrushali C
                70.
                		 Restrict read access to timelineservice v2 data Sub-task Resolved Vrushali C
                71.
                		 [ATSv2] NPE in RM while starting timeline collector on recovery after explicit failover Sub-task Resolved Varun Saxena
                72.
                		 Ensure that supplemented timestamp is stored only for flow run metrics Sub-task Resolved Varun Saxena
                73.
                		 Reader API for sub application entities Sub-task Resolved Rohith Sharma K S
                74.
                		 Supplement timestamp for min start/max end time columns in flow run table to avoid overwrite Sub-task Resolved Vrushali C
                75.
                		 Refactor AppLevelTimelineCollector such that RM does not have aggregator threads created Sub-task Resolved Vrushali C
                76.
                		 Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat Sub-task Resolved Haibo Chen
                77.
                		 [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol Sub-task Resolved Varun Saxena
                78.
                		 Backport HADOOP-14386 to YARN-5355 branch Sub-task Resolved Vrushali C
                79.
                		 [ATSv2] NPE while starting hbase co-processor when HBase authorization is enabled. Sub-task Resolved Rohith Sharma K S
                80.
                		 [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled Sub-task Resolved Varun Saxena
                81.
                		 Nodemanager NPE running jobs with security off Sub-task Resolved Varun Saxena
                82.
                		 Documentation updates for TimelineService v2 Sub-task Resolved Rohith Sharma K S
                83.
                		 Adding a NoOp timeline writer and reader plugin classes for ATSv2 Sub-task Patch Available Prabha Manepalli

                  Activity

                    People

                    • Assignee:
                      vrushalic Vrushali C
                      Reporter:
                      sjlee0 Sangjin Lee
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      21 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: