Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3933

Failures because MALLOC_ARENA_MAX is not set

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 2.0.0-alpha
    • Component/s: mrv2, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We have noticed a bunch of MapReduce test failures on CentOS 6 due to "running beyond virtual memory limits".

      These tests fail with messages of the form:

      [Node Status Updater] nodemanager.NodeStatusUpdaterImpl (NodeStatusUpdaterImpl.java:getNodeStatus(254)) - Sending out status for container: container_id {, app_attempt_id {, application_id {, id: 1, cluster_timestamp: 1330401645767, }, attemptId: 1, }, id: 1, }, state: C_RUNNING, diagnostics: "Container [pid=16750,containerID=container_1330401645767_0001_01_000001] is running beyond virtual memory limits. Current usage: 220.5mb of 2.0gb physical memory used; 7.1gb of 4.2gb virtual memory used. Killing container
      

      The failing tests are:

      TestJobCounters
      TestJobSysDirWithDFS
      TestLazyOutput
      TestMiniMRChildTask
      TestMiniMRClientCluster
      TestReduceFetchFromPartialMem
      TestChild
      TestMapReduceLazyOutput
      TestJobOutputCommitter
      TestMRAppWithCombiner
      TestMRJobs
      TestMRJobsWithHistoryService
      TestMROldApiJobs
      TestSpeculativeExecution
      TestUberAM
      

      I'll upload a patch momentarily.

      1. MAPREDUCE-3933.patch
        1 kB
        Ahmed Radwan
      2. MAPREDUCE-3933_rev2.patch
        1 kB
        Ahmed Radwan

        Issue Links

          Activity

          Hide
          Ahmed Radwan added a comment -

          A patch setting MALLOC_ARENA_MAX in the root pom.xml. I also removed this setting from DistributedShell pom as it is no longer needed when it is added to the root pom. I have verified that all failing tests now run successfully.

          Show
          Ahmed Radwan added a comment - A patch setting MALLOC_ARENA_MAX in the root pom.xml. I also removed this setting from DistributedShell pom as it is no longer needed when it is added to the root pom. I have verified that all failing tests now run successfully.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12516276/MAPREDUCE-3933.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1945//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516276/MAPREDUCE-3933.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1945//console This message is automatically generated.
          Hide
          Ahmed Radwan added a comment -

          Updated patch that modifies the pom under hadoop-project instead of the root pom. Changes to any is resolving the issue but based on an offline discussion with Tucu, he advised to use the pom in hadoop-project, similar to the root pom, any variables set there are inherited by all modules. Thanks Tucu!

          Show
          Ahmed Radwan added a comment - Updated patch that modifies the pom under hadoop-project instead of the root pom. Changes to any is resolving the issue but based on an offline discussion with Tucu, he advised to use the pom in hadoop-project, similar to the root pom, any variables set there are inherited by all modules. Thanks Tucu!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12516495/MAPREDUCE-3933_rev2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1958//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516495/MAPREDUCE-3933_rev2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1958//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          +1

          Show
          Alejandro Abdelnur added a comment - +1
          Hide
          Alejandro Abdelnur added a comment -

          Thanks Ahmed. Committed to trunk and branch-0.23

          Show
          Alejandro Abdelnur added a comment - Thanks Ahmed. Committed to trunk and branch-0.23
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1881 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1881/)
          MAPREDUCE-3933. Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu) (Revision 1295178)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295178
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml
          • /hadoop/common/trunk/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1881 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1881/ ) MAPREDUCE-3933 . Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu) (Revision 1295178) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295178 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #605 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/605/)
          Merge -r 1295177:1295178 from trunk to branch. FIXES: MAPREDUCE-3933 (Revision 1295179)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295179
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml
          • /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #605 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/605/ ) Merge -r 1295177:1295178 from trunk to branch. FIXES: MAPREDUCE-3933 (Revision 1295179) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295179 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #617 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/617/)
          Merge -r 1295177:1295178 from trunk to branch. FIXES: MAPREDUCE-3933 (Revision 1295179)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295179
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml
          • /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #617 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/617/ ) Merge -r 1295177:1295178 from trunk to branch. FIXES: MAPREDUCE-3933 (Revision 1295179) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295179 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1807 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1807/)
          MAPREDUCE-3933. Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu) (Revision 1295178)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295178
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml
          • /hadoop/common/trunk/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1807 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1807/ ) MAPREDUCE-3933 . Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu) (Revision 1295178) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295178 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1815 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1815/)
          MAPREDUCE-3933. Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu) (Revision 1295178)

          Result = ABORTED
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295178
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml
          • /hadoop/common/trunk/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1815 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1815/ ) MAPREDUCE-3933 . Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu) (Revision 1295178) Result = ABORTED tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295178 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #617 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/617/)
          Merge -r 1295177:1295178 from trunk to branch. FIXES: MAPREDUCE-3933 (Revision 1295179)

          Result = ABORTED
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295179
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml
          • /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #617 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/617/ ) Merge -r 1295177:1295178 from trunk to branch. FIXES: MAPREDUCE-3933 (Revision 1295179) Result = ABORTED tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295179 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #971 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/971/)
          MAPREDUCE-3933. Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu) (Revision 1295178)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295178
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml
          • /hadoop/common/trunk/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #971 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/971/ ) MAPREDUCE-3933 . Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu) (Revision 1295178) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295178 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1006 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1006/)
          MAPREDUCE-3933. Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu) (Revision 1295178)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295178
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml
          • /hadoop/common/trunk/hadoop-project/pom.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1006 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1006/ ) MAPREDUCE-3933 . Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu) (Revision 1295178) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295178 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml

            People

            • Assignee:
              Ahmed Radwan
              Reporter:
              Ahmed Radwan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development