XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: api
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We use int32 for memory now, if a cluster has 10k nodes, each node has 210G memory, we will get a negative total cluster memory.

      And another case that easier overflows int32 is: we added all pending resources of running apps to cluster's total pending resources. If a problematic app requires too much resources (let's say 1M+ containers, each of them has 3G containers), int32 will be not enough.

      Even if we can cap each app's pending request, we cannot handle the case that there're many running apps, each of them has capped but still significant numbers of pending resources.

      So we may possibly need to add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource.

        Attachments

        1. YARN-4844.1.patch
          462 kB
          Wangda Tan
        2. YARN-4844.2.patch
          462 kB
          Wangda Tan
        3. YARN-4844.3.patch
          465 kB
          Wangda Tan
        4. YARN-4844.4.patch
          472 kB
          Wangda Tan
        5. YARN-4844.5.patch
          472 kB
          Wangda Tan
        6. YARN-4844.6.patch
          478 kB
          Wangda Tan
        7. YARN-4844.7.patch
          471 kB
          Wangda Tan
        8. YARN-4844.8.patch
          471 kB
          Wangda Tan
        9. YARN-4844.8.branch-2.patch
          466 kB
          Wangda Tan
        10. YARN-4844.9.branch-2.patch
          463 kB
          Wangda Tan
        11. YARN-4844.9.branch
          469 kB
          Wangda Tan
        12. YARN-4844.10.patch
          519 kB
          Wangda Tan
        13. YARN-4844.11.patch
          529 kB
          Wangda Tan
        14. YARN-4844.12.patch
          529 kB
          Wangda Tan
        15. YARN-4844.13.patch
          529 kB
          Wangda Tan
        16. YARN-4844.14.patch
          529 kB
          Wangda Tan
        17. YARN-4844.15.patch
          529 kB
          Wangda Tan
        18. YARN-4844.16.patch
          532 kB
          Wangda Tan
        19. YARN-4844.16.branch-2.patch
          522 kB
          Wangda Tan
        20. YARN-4844-branch-2.addendum.1_.patch
          2 kB
          Wangda Tan
        21. YARN-4844-branch-2.8.0016_.patch
          495 kB
          Wangda Tan
        22. YARN-4844-branch-2.addendum.2.patch
          6 kB
          Wangda Tan
        23. YARN-4844-branch-2.8.addendum.2.patch
          6 kB
          Wangda Tan
        24. YARN-4844.addendum.3.patch
          7 kB
          Wangda Tan
        25. YARN-4844.addendum.4.patch
          5 kB
          Wangda Tan

          Issue Links

            Activity

              People

              • Assignee:
                leftnoteasy Wangda Tan
                Reporter:
                leftnoteasy Wangda Tan
              • Votes:
                0 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: