Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5603

Ability to disable FileInputFormat listLocatedStatus optimization to save client memory

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: 0.23.10, 2.2.0
    • Fix Version/s: None
    • Component/s: client, mrv2
    • Labels:
      None

      Description

      It would be nice if users had the option to disable the listLocatedStatus optimization in FileInputFormat to save client memory.

      1. MAPREDUCE-5603.patch
        15 kB
        Jason Lowe
      2. MAPREDUCE-5603.patch
        15 kB
        Jason Lowe

        Issue Links

          Activity

          Hide
          Jason Lowe added a comment -

          Closing this as won't fix for now. This was originally filed as a stop-gap in case there were any situations where the client needed the extra memory and simply couldn't use a larger heap. However after running with the listLocatedStatus fix for quite some time we've only seen a couple of instances where the client needed more memory to handle the situation and it was an easy fix to simply increase the memory used by the client.

          There's plenty of confs in Hadoop as it is, and I'd rather not add yet another conf if it's not necessary. We can reopen this if the need arises.

          Show
          Jason Lowe added a comment - Closing this as won't fix for now. This was originally filed as a stop-gap in case there were any situations where the client needed the extra memory and simply couldn't use a larger heap. However after running with the listLocatedStatus fix for quite some time we've only seen a couple of instances where the client needed more memory to handle the situation and it was an easy fix to simply increase the memory used by the client. There's plenty of confs in Hadoop as it is, and I'd rather not add yet another conf if it's not necessary. We can reopen this if the need arises.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12614446/MAPREDUCE-5603.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4446//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4446//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614446/MAPREDUCE-5603.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4446//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4446//console This message is automatically generated.
          Hide
          Chen He added a comment -

          +1, patch is good.

          Show
          Chen He added a comment - +1, patch is good.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12614446/MAPREDUCE-5603.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4213//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4213//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614446/MAPREDUCE-5603.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4213//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4213//console This message is automatically generated.
          Hide
          Jason Lowe added a comment -

          Thanks for the review, Kihwal. I think the issues on the Apache build machines has been resolved, so I'm uploading the same patch again to get Jenkins to comment again.

          Show
          Jason Lowe added a comment - Thanks for the review, Kihwal. I think the issues on the Apache build machines has been resolved, so I'm uploading the same patch again to get Jenkins to comment again.
          Hide
          Kihwal Lee added a comment -

          The patch looks good. Since the MAPREDUCE build is broken, can you post your own test result?

          Show
          Kihwal Lee added a comment - The patch looks good. Since the MAPREDUCE build is broken, can you post your own test result?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12611157/MAPREDUCE-5603.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4156//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12611157/MAPREDUCE-5603.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4156//console This message is automatically generated.
          Hide
          Jason Lowe added a comment -

          Patch that adds a mapreduce.input.fileinputformat.uselocatedstatus config to control whether the listLocatedStatus optimization is enabled. The property defaults to true.

          Show
          Jason Lowe added a comment - Patch that adds a mapreduce.input.fileinputformat.uselocatedstatus config to control whether the listLocatedStatus optimization is enabled. The property defaults to true.
          Hide
          Jason Lowe added a comment -

          Sample OOM backtrace for reference:

          Exception in thread "main" java.io.IOException: Failed on local exception:
          java.io.IOException: Error reading responses; Host Details : local host is: "x/x.x.x.x"; destination host is: ""x":x;
              at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:738)
              at org.apache.hadoop.ipc.Client.call(Client.java:1098)
              at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195)
              at com.sun.proxy.$Proxy6.getListing(Unknown Source)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:601)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67)
              at com.sun.proxy.$Proxy6.getListing(Unknown Source)
              at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1286)
              at org.apache.hadoop.hdfs.DistributedFileSystem$1.<init>(DistributedFileSystem.java:418)
              at org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:409)
              at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1654)
              at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:225)
              at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:45)
              at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:265)
              at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:500)
              at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:492)
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:385)
              at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
              at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1264)
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
              at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:573)
              at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:568)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1264)
              at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:568)
              at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:844)
              at x
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
              at x
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:601)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
          Caused by: java.io.IOException: Error reading responses
              at org.apache.hadoop.ipc.Client$Connection.run(Client.java:764)
          Caused by: java.lang.OutOfMemoryError: Java heap space
              at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:64)
              at java.lang.StringBuilder.<init>(StringBuilder.java:97)
              at org.apache.hadoop.io.UTF8.readString(UTF8.java:216)
              at org.apache.hadoop.hdfs.DeprecatedUTF8.readString(DeprecatedUTF8.java:59)
              at org.apache.hadoop.hdfs.protocol.DatanodeID.readFields(DatanodeID.java:212)
              at org.apache.hadoop.hdfs.protocol.DatanodeInfo.readFields(DatanodeInfo.java:389)
              at org.apache.hadoop.hdfs.protocol.LocatedBlock.readFields(LocatedBlock.java:146)
              at org.apache.hadoop.hdfs.protocol.LocatedBlocks.readFields(LocatedBlocks.java:223)
              at org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus.readFields(HdfsLocatedFileStatus.java:87)
              at org.apache.hadoop.hdfs.protocol.DirectoryListing.readFields(DirectoryListing.java:120)
              at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:280)
              at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:75)
              at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:833)
              at org.apache.hadoop.ipc.Client$Connection.run(Client.java:757)
          
          Show
          Jason Lowe added a comment - Sample OOM backtrace for reference: Exception in thread "main" java.io.IOException: Failed on local exception: java.io.IOException: Error reading responses; Host Details : local host is: "x/x.x.x.x"; destination host is: ""x":x; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:738) at org.apache.hadoop.ipc.Client.call(Client.java:1098) at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195) at com.sun.proxy.$Proxy6.getListing(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67) at com.sun.proxy.$Proxy6.getListing(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1286) at org.apache.hadoop.hdfs.DistributedFileSystem$1.<init>(DistributedFileSystem.java:418) at org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:409) at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1654) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:225) at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:45) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:265) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:500) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:492) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:385) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1264) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:573) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:568) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1264) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:568) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:844) at x at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at x at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) Caused by: java.io.IOException: Error reading responses at org.apache.hadoop.ipc.Client$Connection.run(Client.java:764) Caused by: java.lang.OutOfMemoryError: Java heap space at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:64) at java.lang.StringBuilder.<init>(StringBuilder.java:97) at org.apache.hadoop.io.UTF8.readString(UTF8.java:216) at org.apache.hadoop.hdfs.DeprecatedUTF8.readString(DeprecatedUTF8.java:59) at org.apache.hadoop.hdfs.protocol.DatanodeID.readFields(DatanodeID.java:212) at org.apache.hadoop.hdfs.protocol.DatanodeInfo.readFields(DatanodeInfo.java:389) at org.apache.hadoop.hdfs.protocol.LocatedBlock.readFields(LocatedBlock.java:146) at org.apache.hadoop.hdfs.protocol.LocatedBlocks.readFields(LocatedBlocks.java:223) at org.apache.hadoop.hdfs.protocol.HdfsLocatedFileStatus.readFields(HdfsLocatedFileStatus.java:87) at org.apache.hadoop.hdfs.protocol.DirectoryListing.readFields(DirectoryListing.java:120) at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:280) at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:75) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:833) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:757)
          Hide
          Jason Lowe added a comment -

          Recently we ran across a jobclient that failed with an OOM error once we updated the cluster to 0.23.10. The OOM was triggered by the FileInputFormat listLocatedStatus optimization from MAPREDUCE-1981, as the client now caches the BlockLocations of all files along with the FileStatus objects it was caching before. Normally the user can bump the heap size of the client to work around this issue. However if a job has an input with a particularly large number of BlockLocations, as this job did, it would be nice if the user had the option to disable the optimization to reduce the required memory necessary for input split calculations.

          Show
          Jason Lowe added a comment - Recently we ran across a jobclient that failed with an OOM error once we updated the cluster to 0.23.10. The OOM was triggered by the FileInputFormat listLocatedStatus optimization from MAPREDUCE-1981 , as the client now caches the BlockLocations of all files along with the FileStatus objects it was caching before. Normally the user can bump the heap size of the client to work around this issue. However if a job has an input with a particularly large number of BlockLocations, as this job did, it would be nice if the user had the option to disable the optimization to reduce the required memory necessary for input split calculations.

            People

            • Assignee:
              Jason Lowe
              Reporter:
              Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development