Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2928

YARN Timeline Service v.2: alpha 1

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0-alpha1
    • timelineserver
    • None
    • Reviewed
    • Hide
      We are introducing an early preview (alpha 1) of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation.

      YARN Timeline Service v.2 alpha 1 is provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. It should be used only in a test capacity. Most importantly, security is not enabled. Do not set up or use Timeline Service v.2 until security is implemented if security is a critical requirement.

      More details are available in the [YARN Timeline Service v.2](./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html) documentation.
      Show
      We are introducing an early preview (alpha 1) of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation. YARN Timeline Service v.2 alpha 1 is provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. It should be used only in a test capacity. Most importantly, security is not enabled. Do not set up or use Timeline Service v.2 until security is implemented if security is a critical requirement. More details are available in the [YARN Timeline Service v.2](./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html) documentation.

    Description

      We have the application timeline server implemented in yarn per YARN-1530 and YARN-321. Although it is a great feature, we have recognized several critical issues and features that need to be addressed.

      This JIRA proposes the design and implementation changes to address those. This is phase 1 of this effort.

      Attachments

        1. ATSv2.rev1.pdf
          249 kB
          Sangjin Lee
        2. ATSv2.rev2.pdf
          252 kB
          Sangjin Lee
        3. Data model proposal v1.pdf
          89 kB
          Zhijie Shen
        4. Timeline Service Next Gen - Planning - ppt.pptx
          345 kB
          Vinod Kumar Vavilapalli
        5. TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf
          179 kB
          Vrushali C
        6. ATSv2BackendHBaseSchemaproposal.pdf
          259 kB
          Sangjin Lee
        7. timeline_service_v2_next_milestones.pdf
          129 kB
          Sangjin Lee
        8. The YARN Timeline Service v.2 Documentation.pdf
          743 kB
          Sangjin Lee
        9. YARN-2928.01.patch
          2.36 MB
          Sangjin Lee
        10. YARN-2928.02.patch
          2.37 MB
          Sangjin Lee
        11. YARN-2928.03.patch
          2.37 MB
          Sangjin Lee

        Issue Links

          incorporates

          Sub-task - The sub-task of the issue MAPREDUCE-6335 convert load test driver to timeline service v.2

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue MAPREDUCE-6337 add a mode to replay MR job history files to the timeline service

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. YARN-5236 FlowRunCoprocessor brings down HBase RegionServer

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-6331 [Umbrella] Make MapReduce work with Timeline Service Nextgen (YARN-2928)

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Sub-task - The sub-task of the issue YARN-2177 Timeline server web interfaces high-availablity and scalability

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-2520 Scalable and High Available Timeline Server

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-2521 Reliable TimelineClient

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. YARN-1530 [Umbrella] Store, manage and serve per-framework application-timeline data

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. YARN-3538 TimelineServer doesn't catch/translate all exceptions raised

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. YARN-5364 timelineservice modules have indirect dependencies on mapreduce artifacts

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. YARN-5355 YARN Timeline Service v.2: alpha 2

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-3332 [Umbrella] Unified Resource Statistics Collection per node

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. YARN-321 [Umbrella] Generic application history service

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue YARN-2517 Implement TimelineClientAsync

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue YARN-2984 Metrics for container's actual memory usage

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue YARN-2965 Enhance Node Managers to monitor and report the resource usage on machines

          • Major - Major loss of function.
          • Patch Available

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-6732 mapreduce tasks for YARN Timeline Service v.2: alpha 2

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-2141 [Umbrella] Capture container and node resource consumption

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-15686 Add override mechanism for the exempt classes when dynamically loading table coprocessor

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-1012 Report NM aggregated container resource utilization in heartbeat

          • Major - Major loss of function.
          • Resolved
          There are no Sub-Tasks for this issue.

          Activity

            People

              sjlee0 Sangjin Lee
              sjlee0 Sangjin Lee
              Votes:
              1 Vote for this issue
              Watchers:
              91 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: