Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4026

Lower minimum-allocation-mb to sensible defaults

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: mrv2, scheduler
    • Labels:
      None

      Description

      The CapacityScheduler's minimum-allocation-mb is set to 1024.

      The FIFO's minimum-allocation-mb meanwhile, is 128.

      I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.

      1. MAPREDUCE-4026.patch
        9 kB
        Harsh J
      2. MAPREDUCE-4026.patch
        7 kB
        Harsh J
      3. MAPREDUCE-4026.patch
        5 kB
        Harsh J
      4. MAPREDUCE-4026.patch
        5 kB
        Harsh J
      5. MAPREDUCE-4026.patch
        1 kB
        Harsh J

        Issue Links

          Activity

          Hide
          Arun C Murthy added a comment -

          You can't really do much in jvm with 128M heap. Maybe 512M ...

          Show
          Arun C Murthy added a comment - You can't really do much in jvm with 128M heap. Maybe 512M ...
          Hide
          Harsh J added a comment -

          Arun,

          It is only the min-alloc am changing, not the job defaults (mapreduce.map.memory.mb and such) which are presently 1024 or higher, and will need a new JIRA if to be lowered as a default (agree that 128 is too low - either 200 or 512 should be it for those).

          I don't think lowering min alloc default would change default job requirements, but will at least allow folks to pass -D options for memory needs that are lower than 1024 - for more tasks in parallel.

          Or am I missing something?

          Show
          Harsh J added a comment - Arun, It is only the min-alloc am changing, not the job defaults (mapreduce.map.memory.mb and such) which are presently 1024 or higher, and will need a new JIRA if to be lowered as a default (agree that 128 is too low - either 200 or 512 should be it for those). I don't think lowering min alloc default would change default job requirements, but will at least allow folks to pass -D options for memory needs that are lower than 1024 - for more tasks in parallel. Or am I missing something?
          Hide
          Harsh J added a comment -

          Patch that lowers the min-alloc MB to 128 in both existing FIFO and CS schedulers.

          Show
          Harsh J added a comment - Patch that lowers the min-alloc MB to 128 in both existing FIFO and CS schedulers.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12523483/MAPREDUCE-4026.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.yarn.server.TestContainerManagerSecurity
          org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
          org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
          org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
          org.apache.hadoop.mapred.TestClientRedirect
          org.apache.hadoop.mapreduce.security.TestJHSSecurity

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2260//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2260//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523483/MAPREDUCE-4026.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2260//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2260//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Didn't realize there was a hardcoded test in TestFifoScheduler. Fixed test to not hardcode now, and rather reuse a constant.

          Show
          Harsh J added a comment - Didn't realize there was a hardcoded test in TestFifoScheduler. Fixed test to not hardcode now, and rather reuse a constant.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12523491/MAPREDUCE-4026.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2262//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523491/MAPREDUCE-4026.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2262//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Ah earlier patch lacked all sub commits. Re-upping a new one.

          Show
          Harsh J added a comment - Ah earlier patch lacked all sub commits. Re-upping a new one.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12523495/MAPREDUCE-4026.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.yarn.server.TestContainerManagerSecurity
          org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
          org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
          org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
          org.apache.hadoop.mapred.TestClientRedirect
          org.apache.hadoop.mapreduce.security.TestJHSSecurity

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2263//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2263//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523495/MAPREDUCE-4026.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2263//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2263//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Only TestRMWebServices failure seems related to this. Fixed that test to not use hardcodes either. Fixed test passes now.

          The other failures stem out of port binding issues (mis-closed min-YARN clusters?) and a couple of others with kerberos-related issues.

          Show
          Harsh J added a comment - Only TestRMWebServices failure seems related to this. Fixed that test to not use hardcodes either. Fixed test passes now. The other failures stem out of port binding issues (mis-closed min-YARN clusters?) and a couple of others with kerberos-related issues.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12523508/MAPREDUCE-4026.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.yarn.server.TestContainerManagerSecurity
          org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
          org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
          org.apache.hadoop.mapred.TestClientRedirect
          org.apache.hadoop.mapreduce.security.TestJHSSecurity

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2264//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2264//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523508/MAPREDUCE-4026.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2264//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2264//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          I like the patch, I don't really have much of an opinion of the changing of the default values. I only have one NIT and that is with preexisting code. I would like to have MINIMUM_MEMORY and MAXIMUM_MEMORY in both the fifo and the capacity scheduler renamed to DEFAULT_MINIMUM_ALLOCATION and DEFAULT_MAXIMUM_ALLOCATION respectively. This is just so it is less confusing what these values are, especially now that they are exposed to the tests.

          Show
          Robert Joseph Evans added a comment - I like the patch, I don't really have much of an opinion of the changing of the default values. I only have one NIT and that is with preexisting code. I would like to have MINIMUM_MEMORY and MAXIMUM_MEMORY in both the fifo and the capacity scheduler renamed to DEFAULT_MINIMUM_ALLOCATION and DEFAULT_MAXIMUM_ALLOCATION respectively. This is just so it is less confusing what these values are, especially now that they are exposed to the tests.
          Hide
          Harsh J added a comment -

          Bobby - Good point, I should've done that!

          Addressed now, and also fixed some references to the MINIMUM_ALLOCATION config where applicable. MAXIMUM_ALLOCATION doesn't seem to be tested/used for at the moment, and I've filed MAPREDUCE-4171 for that.

          Thanks for taking a look, please do review again!

          To be sure: This only changes minimum allocation values, not job default memory in MR. DistributedShell too in its test uses 128 MB as the min-alloc, for example. So I don't see this change regressing any working features.

          Show
          Harsh J added a comment - Bobby - Good point, I should've done that! Addressed now, and also fixed some references to the MINIMUM_ALLOCATION config where applicable. MAXIMUM_ALLOCATION doesn't seem to be tested/used for at the moment, and I've filed MAPREDUCE-4171 for that. Thanks for taking a look, please do review again! To be sure: This only changes minimum allocation values, not job default memory in MR. DistributedShell too in its test uses 128 MB as the min-alloc, for example. So I don't see this change regressing any working features.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12523606/MAPREDUCE-4026.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.yarn.server.TestContainerManagerSecurity
          org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
          org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
          org.apache.hadoop.mapred.TestClientRedirect
          org.apache.hadoop.mapreduce.security.TestJHSSecurity

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2266//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2266//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523606/MAPREDUCE-4026.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2266//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2266//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Carrying this same effort on MAPREDUCE-3812 along with other relevant and already existing changes there.

          Resolving as duplicate of MAPREDUCE-3812.

          Show
          Harsh J added a comment - Carrying this same effort on MAPREDUCE-3812 along with other relevant and already existing changes there. Resolving as duplicate of MAPREDUCE-3812 .

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development