Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2328

memory-related configurations missing from mapred-default.xml

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.1
    • Component/s: None
    • Labels:
    • Target Version/s:

      Description

      HADOOP-5881 added new configuration parameters for memory-based scheduling, but they weren't added to mapred-default.xml

      1. MAPREDUCE-2328.r1.diff
        1 kB
        Harsh J
      2. MAPREDUCE-2328.r2.diff
        2 kB
        Harsh J
      3. MAPREDUCE-2328.r3.diff
        2 kB
        Harsh J

        Issue Links

          Activity

          Hide
          Harsh J added a comment -

          MAPREDUCE-3223 first needs to go in. Then this may be revisited.

          Show
          Harsh J added a comment - MAPREDUCE-3223 first needs to go in. Then this may be revisited.
          Hide
          Todd Lipcon added a comment -

          Only mapreduce.map.memory.mb and mapreduce.reduce.memory.mb are relevant in 23. The others are now yarn configs and have different names.

          Show
          Todd Lipcon added a comment - Only mapreduce.map.memory.mb and mapreduce.reduce.memory.mb are relevant in 23. The others are now yarn configs and have different names.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-22-branch #85 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/85/)
          MAPREDUCE-2328. Add memory-related configuration parameters to mapred-default.xml. Contributed by Harsh J.

          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190173
          Files :

          • /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/mapreduce/src/java/mapred-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #85 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/85/ ) MAPREDUCE-2328 . Add memory-related configuration parameters to mapred-default.xml. Contributed by Harsh J. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190173 Files : /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt /hadoop/common/branches/branch-0.22/mapreduce/src/java/mapred-default.xml
          Hide
          Harsh J added a comment -

          Hello,

          Heading to rebase this. Would all of these properties still be valid on 0.23?

          Show
          Harsh J added a comment - Hello, Heading to rebase this. Would all of these properties still be valid on 0.23?
          Hide
          Konstantin Shvachko added a comment -

          I committed this to 0.22 branch. Thank you Harsh.
          Leaving it open for the action on trunk / 0.23

          Show
          Konstantin Shvachko added a comment - I committed this to 0.22 branch. Thank you Harsh. Leaving it open for the action on trunk / 0.23
          Hide
          Konstantin Shvachko added a comment -

          Sorry to come even later.
          This does apply to 0.22 and would be good to have it in the release.
          But a patch for trunk is needed.

          Show
          Konstantin Shvachko added a comment - Sorry to come even later. This does apply to 0.22 and would be good to have it in the release. But a patch for trunk is needed.
          Hide
          Arun C Murthy added a comment -

          Sorry to come in late, the patch has gone stale. Can you please rebase? Thanks.

          Show
          Arun C Murthy added a comment - Sorry to come in late, the patch has gone stale. Can you please rebase? Thanks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12483814/MAPREDUCE-2328.r3.diff
          against trunk revision 1139400.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/419//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/419//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/419//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483814/MAPREDUCE-2328.r3.diff against trunk revision 1139400. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/419//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/419//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/419//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Improved explanations for the cluster wide properties.

          Show
          Harsh J added a comment - Improved explanations for the cluster wide properties.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12482154/MAPREDUCE-2328.r2.diff
          against trunk revision 1134180.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.conf.TestNoDefaultsJobConf
          org.apache.hadoop.examples.terasort.TestTeraSort
          org.apache.hadoop.fs.TestDFSIO
          org.apache.hadoop.fs.TestFileSystem
          org.apache.hadoop.ipc.TestSocketFactory
          org.apache.hadoop.mapred.join.TestDatamerge
          org.apache.hadoop.mapred.lib.TestDelegatingInputFormat
          org.apache.hadoop.mapred.pipes.TestPipes
          org.apache.hadoop.mapred.TestBadRecords
          org.apache.hadoop.mapred.TestClusterMapReduceTestCase
          org.apache.hadoop.mapred.TestCommandLineJobSubmission
          org.apache.hadoop.mapred.TestCompressedEmptyMapOutputs
          org.apache.hadoop.mapred.TestControlledMapReduceJob
          org.apache.hadoop.mapred.TestFileInputFormat
          org.apache.hadoop.mapred.TestJobClient
          org.apache.hadoop.mapred.TestJobDirCleanup
          org.apache.hadoop.mapred.TestJobHistory
          org.apache.hadoop.mapred.TestJobName
          org.apache.hadoop.mapred.TestJobQueueInformation
          org.apache.hadoop.mapred.TestJobStatusPersistency
          org.apache.hadoop.mapred.TestJobSysDirWithDFS
          org.apache.hadoop.mapred.TestJobTrackerXmlJsp
          org.apache.hadoop.mapred.TestLazyOutput
          org.apache.hadoop.mapred.TestMapredHeartbeat
          org.apache.hadoop.mapred.TestMapredSystemDir
          org.apache.hadoop.mapred.TestMiniMRChildTask
          org.apache.hadoop.mapred.TestMiniMRClasspath
          org.apache.hadoop.mapred.TestMiniMRDFSCaching
          org.apache.hadoop.mapred.TestMiniMRDFSSort
          org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
          org.apache.hadoop.mapred.TestMiniMRWithDFS
          org.apache.hadoop.mapred.TestMultipleLevelCaching
          org.apache.hadoop.mapred.TestNodeRefresh
          org.apache.hadoop.mapred.TestRecoveryManager
          org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
          org.apache.hadoop.mapred.TestReduceFetch
          org.apache.hadoop.mapred.TestSetupAndCleanupFailure
          org.apache.hadoop.mapred.TestSeveral
          org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
          org.apache.hadoop.mapred.TestSubmitJob
          org.apache.hadoop.mapred.TestTaskFail
          org.apache.hadoop.mapred.TestWebUIAuthorization
          org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
          org.apache.hadoop.mapreduce.lib.input.TestDelegatingInputFormat
          org.apache.hadoop.mapreduce.lib.join.TestJoinDatamerge
          org.apache.hadoop.mapreduce.lib.join.TestJoinProperties
          org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
          org.apache.hadoop.mapreduce.security.TestTokenCacheOldApi
          org.apache.hadoop.mapreduce.security.TestTokenCache
          org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
          org.apache.hadoop.mapreduce.TestMRJobClient
          org.apache.hadoop.security.authorize.TestServiceLevelAuthorization
          org.apache.hadoop.security.TestMapredGroupMappingServiceRefresh
          org.apache.hadoop.tools.TestCopyFiles
          org.apache.hadoop.tools.TestDistCh
          org.apache.hadoop.tools.TestHadoopArchives
          org.apache.hadoop.tools.TestHarFileSystem

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/377//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/377//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/377//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482154/MAPREDUCE-2328.r2.diff against trunk revision 1134180. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.examples.terasort.TestTeraSort org.apache.hadoop.fs.TestDFSIO org.apache.hadoop.fs.TestFileSystem org.apache.hadoop.ipc.TestSocketFactory org.apache.hadoop.mapred.join.TestDatamerge org.apache.hadoop.mapred.lib.TestDelegatingInputFormat org.apache.hadoop.mapred.pipes.TestPipes org.apache.hadoop.mapred.TestBadRecords org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapred.TestCommandLineJobSubmission org.apache.hadoop.mapred.TestCompressedEmptyMapOutputs org.apache.hadoop.mapred.TestControlledMapReduceJob org.apache.hadoop.mapred.TestFileInputFormat org.apache.hadoop.mapred.TestJobClient org.apache.hadoop.mapred.TestJobDirCleanup org.apache.hadoop.mapred.TestJobHistory org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapred.TestJobQueueInformation org.apache.hadoop.mapred.TestJobStatusPersistency org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapred.TestJobTrackerXmlJsp org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestMapredHeartbeat org.apache.hadoop.mapred.TestMapredSystemDir org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapred.TestMiniMRDFSCaching org.apache.hadoop.mapred.TestMiniMRDFSSort org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapred.TestMiniMRWithDFS org.apache.hadoop.mapred.TestMultipleLevelCaching org.apache.hadoop.mapred.TestNodeRefresh org.apache.hadoop.mapred.TestRecoveryManager org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestSetupAndCleanupFailure org.apache.hadoop.mapred.TestSeveral org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapred.TestSubmitJob org.apache.hadoop.mapred.TestTaskFail org.apache.hadoop.mapred.TestWebUIAuthorization org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat org.apache.hadoop.mapreduce.lib.input.TestDelegatingInputFormat org.apache.hadoop.mapreduce.lib.join.TestJoinDatamerge org.apache.hadoop.mapreduce.lib.join.TestJoinProperties org.apache.hadoop.mapreduce.security.TestBinaryTokenFile org.apache.hadoop.mapreduce.security.TestTokenCacheOldApi org.apache.hadoop.mapreduce.security.TestTokenCache org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapreduce.TestMRJobClient org.apache.hadoop.security.authorize.TestServiceLevelAuthorization org.apache.hadoop.security.TestMapredGroupMappingServiceRefresh org.apache.hadoop.tools.TestCopyFiles org.apache.hadoop.tools.TestDistCh org.apache.hadoop.tools.TestHadoopArchives org.apache.hadoop.tools.TestHarFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/377//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/377//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/377//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          New patch with additions and docs improvement based on Todd's comments and links (and mechanisms of how the cluster-wide properties are utilized.)

          Show
          Harsh J added a comment - New patch with additions and docs improvement based on Todd's comments and links (and mechanisms of how the cluster-wide properties are utilized.)
          Hide
          Harsh J added a comment -

          Thanks Todd. I'll put up a fixed patch shortly.

          Show
          Harsh J added a comment - Thanks Todd. I'll put up a fixed patch shortly.
          Hide
          Todd Lipcon added a comment -

          Seems we are still missing mapreduce.map.memory.mb, mapreduce.map.memory.physical.mb, etc from mapred-default.xml.

          I think the descriptions could also be improved. Take a look at src/docs/src/documentation/content/xdocs/cluster_setup.xml in the common repo for pretty good descriptions, as well as this comment.

          Show
          Todd Lipcon added a comment - Seems we are still missing mapreduce.map.memory.mb, mapreduce.map.memory.physical.mb, etc from mapred-default.xml. I think the descriptions could also be improved. Take a look at src/docs/src/documentation/content/xdocs/cluster_setup.xml in the common repo for pretty good descriptions, as well as this comment .
          Hide
          Harsh J added a comment -

          Justification: I believe this is mostly a doc-fix patch, and I've verified that no existing default-loading tests fail, as well as double checked the entered property names for typos/exact matches. The defaults were taken from the code's use of them around MR's code base.

          Show
          Harsh J added a comment - Justification: I believe this is mostly a doc-fix patch, and I've verified that no existing default-loading tests fail, as well as double checked the entered property names for typos/exact matches. The defaults were taken from the code's use of them around MR's code base.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12478782/MAPREDUCE-2328.r1.diff
          against trunk revision 1101741.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/228//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/228//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/228//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478782/MAPREDUCE-2328.r1.diff against trunk revision 1101741. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/228//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/228//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/228//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Patch with brief description of the options, in mapred-default.xml.

          Show
          Harsh J added a comment - Patch with brief description of the options, in mapred-default.xml.

            People

            • Assignee:
              Harsh J
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development