Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-420

OffHeapMemory configuration only supports a single type of executor

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.2

      Description

      At the implementation of the `getMemoryConf` method in the `JobLauncher` class, the method is implemented with an assumption that there is only a single type of container. However, our resources specification file supports not only a single type but multiple types (e.g. transient/reserved or executors with heterogeneous memory specs). I think you could refer to the `requestContainer` method in the `RuntimeMaster` class. This should support the different executors with a uniform ratio or something similar.

      + I think the default value of off-heap memory of 0.2 is a bit too large, as for workloads that do not use the off heap optimization simply leaves 20% of its memory idle. This should be changed in the way that the off-heap memory optimization is used by default or the default number of 0.2 should be minimized.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hwarim Hwarim Hyun
                Reporter:
                wonook Won Wook Song
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m