Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0, 2.0.2-alpha
    • Component/s: mrv1, mrv2, performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This ticket is to add IFile readahead as part of HADOOP-7714.

      1. MAPREDUCE-4511_trunk.patch
        4 kB
        Ahmed Radwan
      2. MAPREDUCE-4511_trunk_rev6.patch
        11 kB
        Ahmed Radwan
      3. MAPREDUCE-4511_trunk_rev5.patch
        9 kB
        Ahmed Radwan
      4. MAPREDUCE-4511_trunk_rev4.patch
        9 kB
        Ahmed Radwan
      5. MAPREDUCE-4511_trunk_rev3.patch
        9 kB
        Ahmed Radwan
      6. MAPREDUCE-4511_trunk_rev2.patch
        9 kB
        Ahmed Radwan
      7. MAPREDUCE-4511_branch1.patch
        4 kB
        Ahmed Radwan
      8. MAPREDUCE-4511_branch-1_rev7.patch
        8 kB
        Ahmed Radwan
      9. MAPREDUCE-4511_branch-1_rev6.patch
        9 kB
        Ahmed Radwan
      10. MAPREDUCE-4511_branch-1_rev5.patch
        7 kB
        Ahmed Radwan
      11. MAPREDUCE-4511_branch-1_rev4.patch
        7 kB
        Ahmed Radwan
      12. MAPREDUCE-4511_branch-1_rev3.patch
        7 kB
        Ahmed Radwan
      13. MAPREDUCE-4511_branch-1_rev2.patch
        7 kB
        Ahmed Radwan

        Issue Links

          Activity

          Hide
          Ahmed Radwan added a comment -

          Here is the updated branch-1 patch based on Todd's HADOOP-7714 patches. Note that this patch requires HADOOP-7754 patch.

          Show
          Ahmed Radwan added a comment - Here is the updated branch-1 patch based on Todd's HADOOP-7714 patches. Note that this patch requires HADOOP-7754 patch.
          Hide
          Ahmed Radwan added a comment -

          Here is the trunk patch. Similarly this patch requires the trunk patch from HADOOP-7754.

          Show
          Ahmed Radwan added a comment - Here is the trunk patch. Similarly this patch requires the trunk patch from HADOOP-7754 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12539253/MAPREDUCE-4511_trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2711//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539253/MAPREDUCE-4511_trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2711//console This message is automatically generated.
          Hide
          Ahmed Radwan added a comment -

          Here are updated versions for both trunk and branch-1 adding the ability to disable and configure the readahead and its length, respectively.

          Show
          Ahmed Radwan added a comment - Here are updated versions for both trunk and branch-1 adding the ability to disable and configure the readahead and its length, respectively.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12539743/MAPREDUCE-4511_branch-1_rev2.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2717//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539743/MAPREDUCE-4511_branch-1_rev2.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2717//console This message is automatically generated.
          Hide
          Ahmed Radwan added a comment -

          Here are the new patches based on HADOOP-7754 updated patches.

          Show
          Ahmed Radwan added a comment - Here are the new patches based on HADOOP-7754 updated patches.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12539961/MAPREDUCE-4511_trunk_rev3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2718//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539961/MAPREDUCE-4511_trunk_rev3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2718//console This message is automatically generated.
          Hide
          Ahmed Radwan added a comment -

          Here are updated patches that address the case when NativeIO is not available (i.e., raPool == null), and also the case when the passed conf to IFileInputStream is null. I saw these issues while conducting further testing.

          Show
          Ahmed Radwan added a comment - Here are updated patches that address the case when NativeIO is not available (i.e., raPool == null), and also the case when the passed conf to IFileInputStream is null. I saw these issues while conducting further testing.
          Hide
          Ahmed Radwan added a comment -

          Here is the trunk version.

          Show
          Ahmed Radwan added a comment - Here is the trunk version.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12540461/MAPREDUCE-4511_trunk_rev4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2723//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540461/MAPREDUCE-4511_trunk_rev4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2723//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          We need HADOOP-7754 in order for this patch to apply and be test-patch-ed, in the mean time, a better name for the method determineFd would be getFileDescriptorIfAvail.

          Show
          Alejandro Abdelnur added a comment - We need HADOOP-7754 in order for this patch to apply and be test-patch-ed, in the mean time, a better name for the method determineFd would be getFileDescriptorIfAvail .
          Hide
          Ahmed Radwan added a comment -

          Here are the updated patches with the new method name.

          Show
          Ahmed Radwan added a comment - Here are the updated patches with the new method name.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12540738/MAPREDUCE-4511_trunk_rev5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2725//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540738/MAPREDUCE-4511_trunk_rev5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2725//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12540738/MAPREDUCE-4511_trunk_rev5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2732//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2732//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540738/MAPREDUCE-4511_trunk_rev5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2732//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2732//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          Looks good, some minor improvements:

          IFileInputStream <init>:

              if (conf != null){
                readahead = conf.getBoolean("mapreduce.ifile.readahead", true);
                readaheadLength = conf.getInt("mapreduce.ifile.readahead.bytes",
                    4 * 1024 * 1024);
              } else {
                readahead = true;
                readaheadLength = 4 * 1024 * 1024;
              }
          

          How about?

          conf = (conf !=null) ? conf : new Configuration();      
          readahead = conf.getBoolean("mapreduce.ifile.readahead", true);
          readaheadLength = conf.getInt("mapreduce.ifile.readahead.bytes",
          

          Instead using literals, we should have constants keys and default in MRConfig.java and have the defaults in mapred-default.xml.

          IFileInputStream.getFileDescriptorIfAvail: Use a local var for FileDescriptor initialized to null, set it in the IF/ELIF and have a single return at the end of the method.

          IFileInputSTream.doReadahead(): the first condition should be 'raPool != null', if there is not readahead pool avail, no point in checking if the job is configured to do it and if there is a FD

          testcase just tests there are no regressions, right? I think it is OK, would not be that easy to test it anyway.

          Show
          Alejandro Abdelnur added a comment - Looks good, some minor improvements: IFileInputStream <init> : if (conf != null ){ readahead = conf.getBoolean( "mapreduce.ifile.readahead" , true ); readaheadLength = conf.getInt( "mapreduce.ifile.readahead.bytes" , 4 * 1024 * 1024); } else { readahead = true ; readaheadLength = 4 * 1024 * 1024; } How about? conf = (conf != null ) ? conf : new Configuration(); readahead = conf.getBoolean( "mapreduce.ifile.readahead" , true ); readaheadLength = conf.getInt( "mapreduce.ifile.readahead.bytes" , Instead using literals, we should have constants keys and default in MRConfig.java and have the defaults in mapred-default.xml. IFileInputStream.getFileDescriptorIfAvail : Use a local var for FileDescriptor initialized to null, set it in the IF/ELIF and have a single return at the end of the method. IFileInputSTream.doReadahead() : the first condition should be 'raPool != null', if there is not readahead pool avail, no point in checking if the job is configured to do it and if there is a FD testcase just tests there are no regressions, right? I think it is OK, would not be that easy to test it anyway.
          Hide
          Ahmed Radwan added a comment -

          Thanks Tucu for the review! I agree about testing. I have updated the patches per your comments.

          Show
          Ahmed Radwan added a comment - Thanks Tucu for the review! I agree about testing. I have updated the patches per your comments.
          Hide
          Alejandro Abdelnur added a comment -

          +1 for patch for trunk, pending jenkins.

          I don't think patch for branch-1 is correct, you should not add those constants to the JobConf as they won't be in trunk/branch-2 thus breaking backwards compatibility. They should be somewhere else, just not in the JobConf.

          Show
          Alejandro Abdelnur added a comment - +1 for patch for trunk, pending jenkins. I don't think patch for branch-1 is correct, you should not add those constants to the JobConf as they won't be in trunk/branch-2 thus breaking backwards compatibility. They should be somewhere else, just not in the JobConf .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541143/MAPREDUCE-4511_trunk_rev6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2741//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2741//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541143/MAPREDUCE-4511_trunk_rev6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2741//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2741//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          test failure seems unrelated, it has been going one for a while already.

          Show
          Alejandro Abdelnur added a comment - test failure seems unrelated, it has been going one for a while already.
          Hide
          Ahmed Radwan added a comment -

          Thanks Tucu, I see you point about keeping the JobConf properties compatible with MR2. I have updated the branch-1 patch to have these constants in the class itself.

          Show
          Ahmed Radwan added a comment - Thanks Tucu, I see you point about keeping the JobConf properties compatible with MR2. I have updated the branch-1 patch to have these constants in the class itself.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541149/MAPREDUCE-4511_branch-1_rev7.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2742//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541149/MAPREDUCE-4511_branch-1_rev7.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2742//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          +1 on rev7 for branch-1.

          Show
          Alejandro Abdelnur added a comment - +1 on rev7 for branch-1.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2583 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2583/)
          MAPREDUCE-4511. Add IFile readahead (ahmed via tucu) (Revision 1373669)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373669
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFileInputStream.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestIFileStreams.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2583 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2583/ ) MAPREDUCE-4511 . Add IFile readahead (ahmed via tucu) (Revision 1373669) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373669 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFileInputStream.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestIFileStreams.java
          Hide
          Alejandro Abdelnur added a comment -

          Thanks Ahmed. Committed to trunk, branch-2 and branch-1.

          Show
          Alejandro Abdelnur added a comment - Thanks Ahmed. Committed to trunk, branch-2 and branch-1.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2648 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2648/)
          MAPREDUCE-4511. Add IFile readahead (ahmed via tucu) (Revision 1373669)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373669
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFileInputStream.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestIFileStreams.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2648 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2648/ ) MAPREDUCE-4511 . Add IFile readahead (ahmed via tucu) (Revision 1373669) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373669 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFileInputStream.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestIFileStreams.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2612 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2612/)
          MAPREDUCE-4511. Add IFile readahead (ahmed via tucu) (Revision 1373669)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373669
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFileInputStream.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestIFileStreams.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2612 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2612/ ) MAPREDUCE-4511 . Add IFile readahead (ahmed via tucu) (Revision 1373669) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373669 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFileInputStream.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestIFileStreams.java
          Hide
          Arun C Murthy added a comment -

          Did we do any performance benchmark and see any improvements?

          Show
          Arun C Murthy added a comment - Did we do any performance benchmark and see any improvements?
          Hide
          Todd Lipcon added a comment -

          I don't know if Ahmed ran benchmarks on the current iteration of the patch, but I saw 10-20% improvement on terasort and some other reduce-heavy customer workloads with this patch on branch-1 back when I originally proposed it. So I don't see why the new implementation wouldn't have similar gains.

          Show
          Todd Lipcon added a comment - I don't know if Ahmed ran benchmarks on the current iteration of the patch, but I saw 10-20% improvement on terasort and some other reduce-heavy customer workloads with this patch on branch-1 back when I originally proposed it. So I don't see why the new implementation wouldn't have similar gains.
          Hide
          Ahmed Radwan added a comment -

          For branch-1&2 last iterations, I have seen 7-10% improvement on terasort and other custom example jobs. The actual % improvement may differ based on the jobs and how heavy their reduce-side is. I have also used the default readahead length in my tests, tuning this parameter for different setups can also have an impact.

          Show
          Ahmed Radwan added a comment - For branch-1&2 last iterations, I have seen 7-10% improvement on terasort and other custom example jobs. The actual % improvement may differ based on the jobs and how heavy their reduce-side is. I have also used the default readahead length in my tests, tuning this parameter for different setups can also have an impact.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1136 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1136/)
          MAPREDUCE-4511. Add IFile readahead (ahmed via tucu) (Revision 1373669)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373669
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFileInputStream.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestIFileStreams.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1136 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1136/ ) MAPREDUCE-4511 . Add IFile readahead (ahmed via tucu) (Revision 1373669) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373669 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFileInputStream.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestIFileStreams.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1168 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1168/)
          MAPREDUCE-4511. Add IFile readahead (ahmed via tucu) (Revision 1373669)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373669
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFileInputStream.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestIFileStreams.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1168 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1168/ ) MAPREDUCE-4511 . Add IFile readahead (ahmed via tucu) (Revision 1373669) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1373669 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFileInputStream.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestIFileStreams.java
          Hide
          Arun C Murthy added a comment -

          Matt - if you don't mind, I'd like to merge this into branch-1.1 since it's been well baked-in. Thoughts?

          Show
          Arun C Murthy added a comment - Matt - if you don't mind, I'd like to merge this into branch-1.1 since it's been well baked-in. Thoughts?
          Hide
          Arun C Murthy added a comment -

          I merged this to branch-1.1 too.

          Show
          Arun C Murthy added a comment - I merged this to branch-1.1 too.
          Hide
          Matt Foley added a comment - - edited

          In branch-1.1, committed 4-line change to this patch, r1392182, to avoid use of interface HasFileDescriptor (which doesn't exist until 1.2.0) in file src/mapred/org/apache/hadoop/mapred/IFileInputStream.java.

          Show
          Matt Foley added a comment - - edited In branch-1.1, committed 4-line change to this patch, r1392182, to avoid use of interface HasFileDescriptor (which doesn't exist until 1.2.0) in file src/mapred/org/apache/hadoop/mapred/IFileInputStream.java.

            People

            • Assignee:
              Ahmed Radwan
              Reporter:
              Ahmed Radwan
            • Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development