Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2349

speed up list[located]status calls from input formats

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.4.0
    • Component/s: task
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      when a job has many input paths - listStatus - or the improved listLocatedStatus - calls (invoked from the getSplits() method) can take a long time. Most of the time is spent waiting for the previous call to complete and then dispatching the next call.

      This can be greatly speeded up by dispatching multiple calls at once (via executors). If the same filesystem client is used - then the calls are much better pipelined (since calls are serialized) and don't impose extra burden on the namenode while at the same time greatly reducing the latency to the client. In a simple test on non-peak hours, this resulted in the getSplits() time reducing from about 3s to about 0.5s.

      1. MAPREDUCE-2349.5.txt
        43 kB
        Siddharth Seth
      2. MAPREDUCE-2349.4.txt
        41 kB
        Siddharth Seth
      3. MAPREDUCE-2349.3.txt
        39 kB
        Siddharth Seth
      4. MAPREDUCE-2349.2.txt
        39 kB
        Siddharth Seth
      5. MAPREDUCE-2349.1.wip.txt
        22 kB
        Siddharth Seth

        Issue Links

          Activity

          Joydeep Sen Sarma created issue -
          Hide
          Liyin Liang added a comment -

          This jira is very meaningful for large, busy cluster.

          Show
          Liyin Liang added a comment - This jira is very meaningful for large, busy cluster.
          Hide
          Siddharth Seth added a comment -

          WIP patch - which changes getSplits to make parallel requests. Have been using this for a while without issues - will try uploading a patch for review soon.

          Show
          Siddharth Seth added a comment - WIP patch - which changes getSplits to make parallel requests. Have been using this for a while without issues - will try uploading a patch for review soon.
          Siddharth Seth made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-2349.1.wip.txt [ 12628972 ]
          Siddharth Seth made changes -
          Target Version/s 2.4.0 [ 12326141 ]
          Siddharth Seth made changes -
          Assignee Siddharth Seth [ sseth ]
          Hide
          Siddharth Seth added a comment -

          Updated patch with some fixes, and unit tests to cover some scenarios. Tests pass with and without changes to FIF. The logic to determine splits (recursive, non-recursive, exceptions etc) is supposed to be no different from what it used to be.

          Show
          Siddharth Seth added a comment - Updated patch with some fixes, and unit tests to cover some scenarios. Tests pass with and without changes to FIF. The logic to determine splits (recursive, non-recursive, exceptions etc) is supposed to be no different from what it used to be.
          Siddharth Seth made changes -
          Attachment MAPREDUCE-2349.2.txt [ 12629825 ]
          Siddharth Seth made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12629825/MAPREDUCE-2349.2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4361//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4361//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4361//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12629825/MAPREDUCE-2349.2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4361//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4361//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4361//console This message is automatically generated.
          Siddharth Seth made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Siddharth Seth added a comment -

          This should fix the findbugs warning.

          Show
          Siddharth Seth added a comment - This should fix the findbugs warning.
          Siddharth Seth made changes -
          Attachment MAPREDUCE-2349.3.txt [ 12630005 ]
          Siddharth Seth made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12630005/MAPREDUCE-2349.3.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4362//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4362//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12630005/MAPREDUCE-2349.3.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4362//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4362//console This message is automatically generated.
          Siddharth Seth made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Siddharth Seth added a comment -

          Updated patch which changes the default to execute the old code path (numThreads=1 and old code).
          The unit tests run through both code paths though.

          Show
          Siddharth Seth added a comment - Updated patch which changes the default to execute the old code path (numThreads=1 and old code). The unit tests run through both code paths though.
          Siddharth Seth made changes -
          Attachment MAPREDUCE-2349.4.txt [ 12634271 ]
          Siddharth Seth made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12634271/MAPREDUCE-2349.4.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4417//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4417//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12634271/MAPREDUCE-2349.4.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4417//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4417//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Started looking at it, nicely done patch! Few minor comments

          • Add the configs to mapred-default.xml as documentation?
          • LIST_STATUS_NUM_THREADS_DEFAULT -> DEFAULT_LIST_STATUS_NUM_THREADS
          • oldListStatus() -> singleThreadedListStatus()
          • Can you add a bit of javadoc to all the new classes and methods in LocatedFileStatusFetcher? Also to the main LocatedFileStatusFetcher class itself.
          • Synchronization needed for ProcessInitialInputPathResult.addError()?
          • Can you group the callable, result and call-back for each type of operation together in two classes?
          • The 'result' variable doesn't need to be a class field of ProcessInputDirCallable. Similarly the one in ProcessInitialInputPathCallable.

          The tests look good!

          Show
          Vinod Kumar Vavilapalli added a comment - Started looking at it, nicely done patch! Few minor comments Add the configs to mapred-default.xml as documentation? LIST_STATUS_NUM_THREADS_DEFAULT -> DEFAULT_LIST_STATUS_NUM_THREADS oldListStatus() -> singleThreadedListStatus() Can you add a bit of javadoc to all the new classes and methods in LocatedFileStatusFetcher? Also to the main LocatedFileStatusFetcher class itself. Synchronization needed for ProcessInitialInputPathResult.addError()? Can you group the callable, result and call-back for each type of operation together in two classes? The 'result' variable doesn't need to be a class field of ProcessInputDirCallable. Similarly the one in ProcessInitialInputPathCallable. The tests look good!
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Siddharth Seth added a comment -

          Thanks for the review. Updated patch attached.

          Add the configs to mapred-default.xml as documentation?

          Done.

          LIST_STATUS_NUM_THREADS_DEFAULT -> DEFAULT_LIST_STATUS_NUM_THREADS

          Done.

          oldListStatus() -> singleThreadedListStatus()

          Done

          Can you add a bit of javadoc to all the new classes and methods in LocatedFileStatusFetcher? Also to the main LocatedFileStatusFetcher class itself.

          Done

          Synchronization needed for ProcessInitialInputPathResult.addError()?

          Not required. It's local to the specific instance.

          Can you group the callable, result and call-back for each type of operation together in two classes?

          Moved the Result into the callable. The CallbackHandler is non static - so moving that requires a fair amount of change, and additional parameters. Have left that as is.

          The 'result' variable doesn't need to be a class field of ProcessInputDirCallable. Similarly the one in ProcessInitialInputPathCallable.

          Made this local to the method.

          Also fixed a typo in one of the log messages

          Show
          Siddharth Seth added a comment - Thanks for the review. Updated patch attached. Add the configs to mapred-default.xml as documentation? Done. LIST_STATUS_NUM_THREADS_DEFAULT -> DEFAULT_LIST_STATUS_NUM_THREADS Done. oldListStatus() -> singleThreadedListStatus() Done Can you add a bit of javadoc to all the new classes and methods in LocatedFileStatusFetcher? Also to the main LocatedFileStatusFetcher class itself. Done Synchronization needed for ProcessInitialInputPathResult.addError()? Not required. It's local to the specific instance. Can you group the callable, result and call-back for each type of operation together in two classes? Moved the Result into the callable. The CallbackHandler is non static - so moving that requires a fair amount of change, and additional parameters. Have left that as is. The 'result' variable doesn't need to be a class field of ProcessInputDirCallable. Similarly the one in ProcessInitialInputPathCallable. Made this local to the method. Also fixed a typo in one of the log messages
          Siddharth Seth made changes -
          Attachment MAPREDUCE-2349.5.txt [ 12635675 ]
          Siddharth Seth made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue relates to MAPREDUCE-5603 [ MAPREDUCE-5603 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Okay, the latest patch looks good. Will check it in when Jenkins says okay..

          Show
          Vinod Kumar Vavilapalli added a comment - Okay, the latest patch looks good. Will check it in when Jenkins says okay..
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12635675/MAPREDUCE-2349.5.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4448//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4448//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12635675/MAPREDUCE-2349.5.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4448//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4448//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Committed this to trunk, branch-2 and branch-2.4. Thanks Sid!

          Show
          Vinod Kumar Vavilapalli added a comment - Committed this to trunk, branch-2 and branch-2.4. Thanks Sid!
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.4.0 [ 12326141 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5362 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5362/)
          MAPREDUCE-2349. Modified FileInputFormat to be able to issue file and block location calls in parallel. Contributed by Siddharth Seth. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579515)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5362 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5362/ ) MAPREDUCE-2349 . Modified FileInputFormat to be able to issue file and block location calls in parallel. Contributed by Siddharth Seth. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579515 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #515 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/515/)
          MAPREDUCE-2349. Modified FileInputFormat to be able to issue file and block location calls in parallel. Contributed by Siddharth Seth. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579515)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #515 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/515/ ) MAPREDUCE-2349 . Modified FileInputFormat to be able to issue file and block location calls in parallel. Contributed by Siddharth Seth. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579515 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1707 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1707/)
          MAPREDUCE-2349. Modified FileInputFormat to be able to issue file and block location calls in parallel. Contributed by Siddharth Seth. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579515)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1707 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1707/ ) MAPREDUCE-2349 . Modified FileInputFormat to be able to issue file and block location calls in parallel. Contributed by Siddharth Seth. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579515 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1732 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1732/)
          MAPREDUCE-2349. Modified FileInputFormat to be able to issue file and block location calls in parallel. Contributed by Siddharth Seth. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579515)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1732 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1732/ ) MAPREDUCE-2349 . Modified FileInputFormat to be able to issue file and block location calls in parallel. Contributed by Siddharth Seth. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579515 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LocatedFileStatusFetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Siddharth Seth
              Reporter:
              Joydeep Sen Sarma
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development