Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2928 YARN Timeline Service v.2: alpha 1
  3. YARN-3816

[Aggregation] App-level aggregation and accumulation for YARN system metrics

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We need application level aggregation of Timeline data:

      • To present end user aggregated states for each application, include: resource (CPU, Memory) consumption across all containers, number of containers launched/completed/failed, etc. We need this for apps while they are running as well as when they are done.
      • Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be aggregated to show details of states in framework level.
      • Other level (Flow/User/Queue) aggregation can be more efficient to be based on Application-level aggregations rather than raw entity-level data as much less raws need to scan (with filter out non-aggregated entities, like: events, configurations, etc.).

        Attachments

        1. Application Level Aggregation of Timeline Data.pdf
          124 kB
          Junping Du
        2. YARN-3816-feature-YARN-2928.v4.1.patch
          65 kB
          Junping Du
        3. YARN-3816-poc-v1.patch
          41 kB
          Junping Du
        4. YARN-3816-poc-v2.patch
          53 kB
          Junping Du
        5. YARN-3816-YARN-2928-v1.patch
          44 kB
          Junping Du
        6. YARN-3816-YARN-2928-v2.1.patch
          63 kB
          Junping Du
        7. YARN-3816-YARN-2928-v2.2.patch
          67 kB
          Junping Du
        8. YARN-3816-YARN-2928-v2.3.patch
          67 kB
          Junping Du
        9. YARN-3816-YARN-2928-v2.patch
          60 kB
          Junping Du
        10. YARN-3816-YARN-2928-v3.1.patch
          67 kB
          Junping Du
        11. YARN-3816-YARN-2928-v3.patch
          67 kB
          Junping Du
        12. YARN-3816-YARN-2928-v4.patch
          67 kB
          Junping Du
        13. YARN-3816-YARN-2928-v5.patch
          62 kB
          Li Lu
        14. YARN-3816-YARN-2928-v6.patch
          54 kB
          Li Lu
        15. YARN-3816-YARN-2928-v7.patch
          48 kB
          Li Lu
        16. YARN-3816-YARN-2928-v8.patch
          51 kB
          Li Lu
        17. YARN-3816-YARN-2928-v9.patch
          54 kB
          Li Lu

          Issue Links

            Activity

              People

              • Assignee:
                gtCarrera9 Li Lu
                Reporter:
                djp Junping Du
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: