Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2928 YARN Timeline Service v.2: alpha 1
  3. YARN-3816

[Aggregation] App-level aggregation and accumulation for YARN system metrics

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      We need application level aggregation of Timeline data:

      • To present end user aggregated states for each application, include: resource (CPU, Memory) consumption across all containers, number of containers launched/completed/failed, etc. We need this for apps while they are running as well as when they are done.
      • Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be aggregated to show details of states in framework level.
      • Other level (Flow/User/Queue) aggregation can be more efficient to be based on Application-level aggregations rather than raw entity-level data as much less raws need to scan (with filter out non-aggregated entities, like: events, configurations, etc.).

      Attachments

        1. YARN-3816-YARN-2928-v9.patch
          54 kB
          Li Lu
        2. YARN-3816-YARN-2928-v8.patch
          51 kB
          Li Lu
        3. YARN-3816-YARN-2928-v7.patch
          48 kB
          Li Lu
        4. YARN-3816-YARN-2928-v6.patch
          54 kB
          Li Lu
        5. YARN-3816-YARN-2928-v5.patch
          62 kB
          Li Lu
        6. YARN-3816-YARN-2928-v4.patch
          67 kB
          Junping Du
        7. YARN-3816-YARN-2928-v3.patch
          67 kB
          Junping Du
        8. YARN-3816-YARN-2928-v3.1.patch
          67 kB
          Junping Du
        9. YARN-3816-YARN-2928-v2.patch
          60 kB
          Junping Du
        10. YARN-3816-YARN-2928-v2.3.patch
          67 kB
          Junping Du
        11. YARN-3816-YARN-2928-v2.2.patch
          67 kB
          Junping Du
        12. YARN-3816-YARN-2928-v2.1.patch
          63 kB
          Junping Du
        13. YARN-3816-YARN-2928-v1.patch
          44 kB
          Junping Du
        14. YARN-3816-poc-v2.patch
          53 kB
          Junping Du
        15. YARN-3816-poc-v1.patch
          41 kB
          Junping Du
        16. YARN-3816-feature-YARN-2928.v4.1.patch
          65 kB
          Junping Du
        17. Application Level Aggregation of Timeline Data.pdf
          124 kB
          Junping Du

        Issue Links

          Activity

            People

              gtcarrera9 Li Lu
              junping_du Junping Du
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: