Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2928

YARN Timeline Service v.2: alpha 1

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0-alpha1
    • timelineserver
    • None
    • Reviewed
    • Hide
      We are introducing an early preview (alpha 1) of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation.

      YARN Timeline Service v.2 alpha 1 is provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. It should be used only in a test capacity. Most importantly, security is not enabled. Do not set up or use Timeline Service v.2 until security is implemented if security is a critical requirement.

      More details are available in the [YARN Timeline Service v.2](./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html) documentation.
      Show
      We are introducing an early preview (alpha 1) of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation. YARN Timeline Service v.2 alpha 1 is provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. It should be used only in a test capacity. Most importantly, security is not enabled. Do not set up or use Timeline Service v.2 until security is implemented if security is a critical requirement. More details are available in the [YARN Timeline Service v.2](./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html) documentation.

    Description

      We have the application timeline server implemented in yarn per YARN-1530 and YARN-321. Although it is a great feature, we have recognized several critical issues and features that need to be addressed.

      This JIRA proposes the design and implementation changes to address those. This is phase 1 of this effort.

      Attachments

        1. ATSv2.rev1.pdf
          249 kB
          Sangjin Lee
        2. ATSv2.rev2.pdf
          252 kB
          Sangjin Lee
        3. ATSv2BackendHBaseSchemaproposal.pdf
          259 kB
          Sangjin Lee
        4. Data model proposal v1.pdf
          89 kB
          Zhijie Shen
        5. The YARN Timeline Service v.2 Documentation.pdf
          743 kB
          Sangjin Lee
        6. timeline_service_v2_next_milestones.pdf
          129 kB
          Sangjin Lee
        7. Timeline Service Next Gen - Planning - ppt.pptx
          345 kB
          Vinod Kumar Vavilapalli
        8. TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf
          179 kB
          Vrushali C
        9. YARN-2928.01.patch
          2.36 MB
          Sangjin Lee
        10. YARN-2928.02.patch
          2.37 MB
          Sangjin Lee
        11. YARN-2928.03.patch
          2.37 MB
          Sangjin Lee

        Issue Links

          incorporates

          Sub-task - The sub-task of the issue MAPREDUCE-6335 convert load test driver to timeline service v.2

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue MAPREDUCE-6337 add a mode to replay MR job history files to the timeline service

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. YARN-5236 FlowRunCoprocessor brings down HBase RegionServer

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-6331 [Umbrella] Make MapReduce work with Timeline Service Nextgen (YARN-2928)

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Sub-task - The sub-task of the issue YARN-2177 Timeline server web interfaces high-availablity and scalability

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-2520 Scalable and High Available Timeline Server

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-2521 Reliable TimelineClient

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. YARN-1530 [Umbrella] Store, manage and serve per-framework application-timeline data

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. YARN-3538 TimelineServer doesn't catch/translate all exceptions raised

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. YARN-5364 timelineservice modules have indirect dependencies on mapreduce artifacts

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. YARN-5355 YARN Timeline Service v.2: alpha 2

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-3332 [Umbrella] Unified Resource Statistics Collection per node

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. YARN-321 [Umbrella] Generic application history service

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue YARN-2517 Implement TimelineClientAsync

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-2984 Metrics for container's actual memory usage

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue YARN-2965 Enhance Node Managers to monitor and report the resource usage on machines

          • Major - Major loss of function.
          • Patch Available

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-6732 mapreduce tasks for YARN Timeline Service v.2: alpha 2

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-2141 [Umbrella] Capture container and node resource consumption

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-15686 Add override mechanism for the exempt classes when dynamically loading table coprocessor

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-1012 Report NM aggregated container resource utilization in heartbeat

          • Major - Major loss of function.
          • Resolved
          There are no Sub-Tasks for this issue.

          Activity

            People

              sjlee0 Sangjin Lee
              sjlee0 Sangjin Lee
              Votes:
              1 Vote for this issue
              Watchers:
              91 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: