Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5059

'whoami', 'topologyscript' calls failing with java.io.IOException: error=12, Cannot allocate memory

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: util
    • Labels:
      None
    • Environment:

      On nodes with
      physical memory 32G
      Swap 16G

      Primary/Secondary Namenode using 25G of heap or more

      Description

      We've seen primary/secondary namenodes fail when calling whoami or topologyscripts.
      (Discussed as part of HADOOP-4998)

      Sample stack traces.

      Primary Namenode

      2009-01-12 03:57:27,381 WARN org.apache.hadoop.net.ScriptBasedMapping: java.io.IOException: Cannot run program
      "/path/topologyProgram" (in directory "/path"):
      java.io.IOException: error=12, Cannot allocate memory
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
              at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
              at org.apache.hadoop.util.Shell.run(Shell.java:134)
              at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:286)
              at org.apache.hadoop.net.ScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:122)
              at org.apache.hadoop.net.ScriptBasedMapping.resolve(ScriptBasedMapping.java:73)
              at org.apache.hadoop.dfs.FSNamesystem$ResolutionMonitor.run(FSNamesystem.java:1869)
              at java.lang.Thread.run(Thread.java:619)
      Caused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory
              at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
              at java.lang.ProcessImpl.start(ProcessImpl.java:65)
              at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
              ... 7 more
      
      2009-01-12 03:57:27,381 ERROR org.apache.hadoop.fs.FSNamesystem: The resolve call returned null! Using /default-rack
      for some hosts
      2009-01-12 03:57:27,381 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/55.5.55.55:50010
      
      

      Secondary Namenode

      2008-10-09 02:00:58,288 ERROR org.apache.hadoop.dfs.NameNode.Secondary: java.io.IOException:
      javax.security.auth.login.LoginException: Login failed: Cannot run program "whoami": java.io.IOException:
      error=12, Cannot allocate memory
              at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
              at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
              at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:257)
              at org.apache.hadoop.dfs.FSNamesystem.setConfigurationParameters(FSNamesystem.java:370)
              at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:359)
              at org.apache.hadoop.dfs.SecondaryNameNode.doMerge(SecondaryNameNode.java:340)
              at org.apache.hadoop.dfs.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:312)
              at org.apache.hadoop.dfs.SecondaryNameNode.run(SecondaryNameNode.java:223)
              at java.lang.Thread.run(Thread.java:619)
      
              at org.apache.hadoop.dfs.FSNamesystem.setConfigurationParameters(FSNamesystem.java:372)
              at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:359)
              at org.apache.hadoop.dfs.SecondaryNameNode.doMerge(SecondaryNameNode.java:340)
              at org.apache.hadoop.dfs.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:312)
              at org.apache.hadoop.dfs.SecondaryNameNode.run(SecondaryNameNode.java:223)
              at java.lang.Thread.run(Thread.java:619)
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                knoguchi Koji Noguchi
              • Votes:
                0 Vote for this issue
                Watchers:
                28 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: