Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I'm looking now at my jobtracker's list of running reduce tasks. One of them is 120.05% complete, the other is 107.28% complete.

      I understand that these numbers are estimates, but there is no case in which an estimate of 100% for a non-complete task is better than an estimate of 99.99%, nor is there any case in which an estimate greater than 100% is valid.

      I suggest that whatever logic is computing these set 99.99% as a hard maximum.

      1. more than 100%.bmp
        1.15 MB
        Devaraj K
      2. MAPREDUCE-2264-trunk-addendum.patch
        2 kB
        Sandy Ryza
      3. MAPREDUCE-2264-trunk-5.patch
        24 kB
        Sandy Ryza
      4. MAPREDUCE-2264-trunk-5.patch
        24 kB
        Sandy Ryza
      5. MAPREDUCE-2264-trunk-4.patch
        24 kB
        Sandy Ryza
      6. MAPREDUCE-2264-trunk-3.patch
        19 kB
        Sandy Ryza
      7. MAPREDUCE-2264-trunk-2.patch
        18 kB
        Sandy Ryza
      8. MAPREDUCE-2264-trunk-1.patch
        18 kB
        Sandy Ryza
      9. MAPREDUCE-2264-trunk-1.patch
        18 kB
        Sandy Ryza
      10. MAPREDUCE-2264-trunk.patch
        11 kB
        Devaraj K
      11. MAPREDUCE-2264-branch-1-2.patch
        15 kB
        Sandy Ryza
      12. MAPREDUCE-2264-branch-1-1.patch
        15 kB
        Sandy Ryza
      13. MAPREDUCE-2264-branch-1.patch
        14 kB
        Sandy Ryza
      14. MAPREDUCE-2264-0.20.3.patch
        9 kB
        Devaraj K
      15. MAPREDUCE-2264-0.20.205-1.patch
        14 kB
        Devaraj K
      16. MAPREDUCE-2264-0.20.205.patch
        9 kB
        Devaraj K

        Issue Links

          Activity

          Adam Kramer created issue -
          Hide
          Liyin Liang added a comment -

          I think HADOOP-5210 has fixed this bug.

          Show
          Liyin Liang added a comment - I think HADOOP-5210 has fixed this bug.
          Hide
          Ravi Gummadi added a comment -

          Is this an issue at job level progress only ? Are task progress percentages for all tasks under 100% ? If a particular task's progress is shown as > 100%, please provide the info of which phase that task is in when you see progress > 100%.

          Show
          Ravi Gummadi added a comment - Is this an issue at job level progress only ? Are task progress percentages for all tasks under 100% ? If a particular task's progress is shown as > 100%, please provide the info of which phase that task is in when you see progress > 100%.
          Hide
          Ravi Gummadi added a comment -

          BTW, Which version of hadoop are you using ? Does it already include the fixes/patches of HADOOP-5210 and HADOOP-5572 ?

          Show
          Ravi Gummadi added a comment - BTW, Which version of hadoop are you using ? Does it already include the fixes/patches of HADOOP-5210 and HADOOP-5572 ?
          Jeff Hammerbacher made changes -
          Field Original Value New Value
          Link This issue is related to HADOOP-5210 [ HADOOP-5210 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to HADOOP-5572 [ HADOOP-5572 ]
          Hide
          Devaraj K added a comment -

          Hi Ravi,

          I am also facing the same problem. Reduce tasks progress is going more than 100% when the reduce input size is high.

          Merger.java
              private void adjustPriorityQueue(Segment<K, V> reader) throws IOException{
                long startPos = reader.getPosition();
                boolean hasNext = reader.next();
                long endPos = reader.getPosition();
                totalBytesProcessed += endPos - startPos;
                mergeProgress.set(totalBytesProcessed * progPerByte);
                if (hasNext) {
                  adjustTop();
                } else {
                  pop();
                  reader.close();
                }

          Here the totalBytesProcessed is becoming more than the totalBytes and (totalBytesProcessed * progPerByte) is coming more than 1.

          I am using the 0.20.2 version and I have HADOOP-5210 ,HADOOP-5572 patches also in the code. Please check the screenshot attached.

          Show
          Devaraj K added a comment - Hi Ravi, I am also facing the same problem. Reduce tasks progress is going more than 100% when the reduce input size is high. Merger.java private void adjustPriorityQueue(Segment<K, V> reader) throws IOException{ long startPos = reader.getPosition(); boolean hasNext = reader.next(); long endPos = reader.getPosition(); totalBytesProcessed += endPos - startPos; mergeProgress.set(totalBytesProcessed * progPerByte); if (hasNext) { adjustTop(); } else { pop(); reader.close(); } Here the totalBytesProcessed is becoming more than the totalBytes and (totalBytesProcessed * progPerByte) is coming more than 1. I am using the 0.20.2 version and I have HADOOP-5210 , HADOOP-5572 patches also in the code. Please check the screenshot attached.
          Devaraj K made changes -
          Attachment more than 100%.bmp [ 12474308 ]
          Hide
          Ravi Gummadi added a comment -

          Is map output compression turned ON ?

          Show
          Ravi Gummadi added a comment - Is map output compression turned ON ?
          Hide
          Devaraj K added a comment -

          yes, "mapred.compress.map.output" is true.

          Show
          Devaraj K added a comment - yes, "mapred.compress.map.output" is true.
          Hide
          Ravi Gummadi added a comment -

          Hmm. Then that may not be considered/handled properly when adding up sizes of segments to get totalBytes ? The size of segment could be compressed size. But the totalBytesProcessed seems to be uncompressed size.

          Show
          Ravi Gummadi added a comment - Hmm. Then that may not be considered/handled properly when adding up sizes of segments to get totalBytes ? The size of segment could be compressed size. But the totalBytesProcessed seems to be uncompressed size.
          Devaraj K made changes -
          Attachment MAPREDUCE-2264-0.20.3.patch [ 12479913 ]
          Attachment MAPREDUCE-2264-trunk.patch [ 12479914 ]
          Devaraj K made changes -
          Assignee Devaraj K [ devaraj.k ]
          Hide
          Devaraj K added a comment -

          It was considering compressed bytes size for calculating the totalBytes because of this it is showing the progress as more than 100%.
          Provided patch consider the uncompressed size of the data for calculating the progress.

          Show
          Devaraj K added a comment - It was considering compressed bytes size for calculating the totalBytes because of this it is showing the progress as more than 100%. Provided patch consider the uncompressed size of the data for calculating the progress.
          Devaraj K made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.20.4 [ 12316034 ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479914/MAPREDUCE-2264-trunk.patch
          against trunk revision 1125428.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.mapred.TestReduceFetch
          org.apache.hadoop.mapreduce.TestMapReduceLocal

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/286//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/286//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/286//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479914/MAPREDUCE-2264-trunk.patch against trunk revision 1125428. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapreduce.TestMapReduceLocal +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/286//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/286//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/286//console This message is automatically generated.
          Owen O'Malley made changes -
          Fix Version/s 0.20.4 [ 12316034 ]
          Hide
          Arun C Murthy added a comment -

          Sorry, patch has gone stale. Can you pls rebase it? Tx

          Show
          Arun C Murthy added a comment - Sorry, patch has gone stale. Can you pls rebase it? Tx
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Devaraj K added a comment -

          It is not applicable for mrv2. Provided patch for 0.20.205 version.

          Show
          Devaraj K added a comment - It is not applicable for mrv2. Provided patch for 0.20.205 version.
          Devaraj K made changes -
          Attachment MAPREDUCE-2264-0.20.205.patch [ 12493349 ]
          Devaraj K made changes -
          Fix Version/s 0.20.205.0 [ 12316391 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Affects Version/s 0.20.2 [ 12314205 ]
          Affects Version/s 0.20.205.0 [ 12316391 ]
          Devaraj K made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493349/MAPREDUCE-2264-0.20.205.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/640//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493349/MAPREDUCE-2264-0.20.205.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/640//console This message is automatically generated.
          Hide
          Devaraj K added a comment -

          Updated patch is intended for 0.20.205 version.

          Show
          Devaraj K added a comment - Updated patch is intended for 0.20.205 version.
          Hide
          Ravi Gummadi added a comment -

          Had a quick look at the patch. Code changes look fine to me.

          Devaraj, Did you test this patch with your job(which was giving >100% reduceTaskProgress earlier) ?
          Would it be possible for you to add a unit test for this fix ?

          Show
          Ravi Gummadi added a comment - Had a quick look at the patch. Code changes look fine to me. Devaraj, Did you test this patch with your job(which was giving >100% reduceTaskProgress earlier) ? Would it be possible for you to add a unit test for this fix ?
          Hide
          Devaraj K added a comment -

          Thanks Ravi for looking into the patch.

          I tested this patch with the job which was going beyond 100%.
          For the job with two reducers and input data as 150 GB, before this patch it was going around 300% and after applying this patch it worked fine.
          I will try to add unit tests for this.

          Show
          Devaraj K added a comment - Thanks Ravi for looking into the patch. I tested this patch with the job which was going beyond 100%. For the job with two reducers and input data as 150 GB, before this patch it was going around 300% and after applying this patch it worked fine. I will try to add unit tests for this.
          Hide
          Arun C Murthy added a comment -

          Cancelling since Devaraj is still adding unit tests.

          Show
          Arun C Murthy added a comment - Cancelling since Devaraj is still adding unit tests.
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Devaraj K added a comment -

          Sorry for late response.

          Updated the patch with unit test.

          Show
          Devaraj K added a comment - Sorry for late response. Updated the patch with unit test.
          Devaraj K made changes -
          Attachment MAPREDUCE-2264-0.20.205-1.patch [ 12494838 ]
          Devaraj K made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12494838/MAPREDUCE-2264-0.20.205-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/768//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494838/MAPREDUCE-2264-0.20.205-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/768//console This message is automatically generated.
          Matt Foley made changes -
          Fix Version/s 0.22.0 [ 12314184 ]
          Fix Version/s 0.20.205.0 [ 12316391 ]
          Target Version/s 0.20.206.0, 0.22.0 [ 12317960, 12314184 ]
          Arun C Murthy made changes -
          Labels critical-0.22.0
          Matt Foley made changes -
          Target Version/s 0.22.0, 1.1.0 [ 12314184, 12317960 ] 0.22.0, 1.1.1 [ 12314184, 12321660 ]
          Sandy Ryza made changes -
          Attachment MAPREDUCE-2264-trunk-1.patch [ 12553396 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12553396/MAPREDUCE-2264-trunk-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3028//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3028//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553396/MAPREDUCE-2264-trunk-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3028//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3028//console This message is automatically generated.
          Hide
          Matt Foley added a comment -

          moved target version to 1.2.0 upon publishing 1.1.1 RC.

          Show
          Matt Foley added a comment - moved target version to 1.2.0 upon publishing 1.1.1 RC.
          Matt Foley made changes -
          Target Version/s 0.22.0, 1.1.1 [ 12314184, 12321660 ] 0.22.0, 1.2.0 [ 12314184, 12321661 ]
          Sandy Ryza made changes -
          Attachment MAPREDUCE-2264-branch-1.patch [ 12564529 ]
          Hide
          Sandy Ryza added a comment -

          Uploaded a refresh patch for branch-1

          Show
          Sandy Ryza added a comment - Uploaded a refresh patch for branch-1
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12564529/MAPREDUCE-2264-branch-1.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3233//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12564529/MAPREDUCE-2264-branch-1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3233//console This message is automatically generated.
          Sandy Ryza made changes -
          Attachment MAPREDUCE-2264-trunk-1.patch [ 12564531 ]
          Hide
          Sandy Ryza added a comment -

          Also uploaded a refresh patch for trunk

          Show
          Sandy Ryza added a comment - Also uploaded a refresh patch for trunk
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12564531/MAPREDUCE-2264-trunk-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3234//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3234//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12564531/MAPREDUCE-2264-trunk-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3234//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3234//console This message is automatically generated.
          Sandy Ryza made changes -
          Attachment MAPREDUCE-2264-trunk-2.patch [ 12566371 ]
          Hide
          Sandy Ryza added a comment -

          Uploaded a refresh patch post-pluggable merge/shuffle.

          Show
          Sandy Ryza added a comment - Uploaded a refresh patch post-pluggable merge/shuffle.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12566371/MAPREDUCE-2264-trunk-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3267//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3267//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566371/MAPREDUCE-2264-trunk-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3267//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3267//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment - - edited

          patch looks good. Could we, as part of the test, test that for uncompressed data we also reach 100%?

          Show
          Alejandro Abdelnur added a comment - - edited patch looks good. Could we, as part of the test, test that for uncompressed data we also reach 100%?
          Sandy Ryza made changes -
          Attachment MAPREDUCE-2264-trunk-3.patch [ 12566411 ]
          Hide
          Sandy Ryza added a comment -

          Uploaded a patch that adds a test for the calculation when data is not compressed.

          Show
          Sandy Ryza added a comment - Uploaded a patch that adds a test for the calculation when data is not compressed.
          Hide
          Alejandro Abdelnur added a comment -

          +1 (run latest patch locally, only change is adding a new test method)

          Show
          Alejandro Abdelnur added a comment - +1 (run latest patch locally, only change is adding a new test method)
          Hide
          Alejandro Abdelnur added a comment -

          Thanks DevarajK and Sandy. Committed to trunk. For branch-2 we need MAPREDUCE-4049, MAPREDUCE-4809, MAPREDUCE-4807 & MAPREDUCE-4808 and this patch will apply cleanly there.

          Show
          Alejandro Abdelnur added a comment - Thanks DevarajK and Sandy. Committed to trunk. For branch-2 we need MAPREDUCE-4049 , MAPREDUCE-4809 , MAPREDUCE-4807 & MAPREDUCE-4808 and this patch will apply cleanly there.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12566411/MAPREDUCE-2264-trunk-3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3270//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3270//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566411/MAPREDUCE-2264-trunk-3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3270//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3270//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3279 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3279/)
          MAPREDUCE-2264. Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1438277)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1438277
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3279 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3279/ ) MAPREDUCE-2264 . Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1438277) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1438277 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Sandy Ryza made changes -
          Attachment MAPREDUCE-2264-branch-1-1.patch [ 12566415 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12566415/MAPREDUCE-2264-branch-1-1.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3271//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566415/MAPREDUCE-2264-branch-1-1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3271//console This message is automatically generated.
          Sandy Ryza made changes -
          Attachment MAPREDUCE-2264-branch-1-2.patch [ 12566420 ]
          Hide
          Alejandro Abdelnur added a comment -

          thanks again. committed to branch-1.

          Show
          Alejandro Abdelnur added a comment - thanks again. committed to branch-1.
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 1.2.0 [ 12321661 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #107 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/107/)
          MAPREDUCE-2264. Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1438277)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1438277
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #107 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/107/ ) MAPREDUCE-2264 . Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1438277) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1438277 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1296 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1296/)
          MAPREDUCE-2264. Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1438277)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1438277
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1296 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1296/ ) MAPREDUCE-2264 . Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1438277) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1438277 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1324 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1324/)
          MAPREDUCE-2264. Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1438277)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1438277
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1324 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1324/ ) MAPREDUCE-2264 . Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1438277) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1438277 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Alejandro Abdelnur made changes -
          Fix Version/s 2.0.3-alpha [ 12323275 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Hide
          Chris Douglas added a comment -

          The patch to trunk changes onDiskMapOutputs to be a TreeSet of CompressAwarePath instead of Path, but the latter doesn't implement Comparable, neither is the TreeSet instantiated with a Comparator. So there's no defined ordering for the sorted set.

          Show
          Chris Douglas added a comment - The patch to trunk changes onDiskMapOutputs to be a TreeSet of CompressAwarePath instead of Path , but the latter doesn't implement Comparable , neither is the TreeSet instantiated with a Comparator . So there's no defined ordering for the sorted set.
          Sandy Ryza made changes -
          Link This issue is related to MAPREDUCE-4965 [ MAPREDUCE-4965 ]
          Hide
          Sandy Ryza added a comment -

          Filed MAPREDUCE-4965 to fix this.

          Show
          Sandy Ryza added a comment - Filed MAPREDUCE-4965 to fix this.
          Hide
          Arun C Murthy added a comment -

          Alternately, we could re-open this jira and fix it.

          Show
          Arun C Murthy added a comment - Alternately, we could re-open this jira and fix it.
          Hide
          Alejandro Abdelnur added a comment -

          Agree with Arun. Reverting from trunk and branch-2. Sandy, please take care of the proper fix here. I'm wondering why none of the existing testcases caught this.

          Show
          Alejandro Abdelnur added a comment - Agree with Arun. Reverting from trunk and branch-2. Sandy, please take care of the proper fix here. I'm wondering why none of the existing testcases caught this.
          Alejandro Abdelnur made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3287 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3287/)
          Revering MAPREDUCE-2264 (Revision 1439561)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1439561
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3287 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3287/ ) Revering MAPREDUCE-2264 (Revision 1439561) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1439561 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Hide
          Arun C Murthy added a comment -

          We could just fix as an addendum too.

          Show
          Arun C Murthy added a comment - We could just fix as an addendum too.
          Hide
          Alejandro Abdelnur added a comment -

          too late, reverted already.

          Show
          Alejandro Abdelnur added a comment - too late, reverted already.
          Sandy Ryza made changes -
          Attachment MAPREDUCE-2264-trunk-4.patch [ 12566834 ]
          Hide
          Sandy Ryza added a comment -

          The latest patch fixes the issue and adds a test to exercise the code that broke.

          Show
          Sandy Ryza added a comment - The latest patch fixes the issue and adds a test to exercise the code that broke.
          Hide
          Ravi Prakash added a comment -

          I also wanted to point out MAPREDUCE-3685 which identified some other problems in MergeManger right now.

          Show
          Ravi Prakash added a comment - I also wanted to point out MAPREDUCE-3685 which identified some other problems in MergeManger right now.
          Ravi Prakash made changes -
          Link This issue duplicates MAPREDUCE-3267 [ MAPREDUCE-3267 ]
          Hide
          Arun C Murthy added a comment -

          Thanks Sandy... reviewing patch now.

          Show
          Arun C Murthy added a comment - Thanks Sandy... reviewing patch now.
          Hide
          Alejandro Abdelnur added a comment -

          the latest patch looks good, it has a couple of typos in the testcase 'segement'. Chris thanks for catching that. Arun, looks good to you now?

          Show
          Alejandro Abdelnur added a comment - the latest patch looks good, it has a couple of typos in the testcase 'segement'. Chris thanks for catching that. Arun, looks good to you now?
          Sandy Ryza made changes -
          Attachment MAPREDUCE-2264-trunk-5.patch [ 12566864 ]
          Hide
          Sandy Ryza added a comment -

          Uploaded a patch that fixes the typo, removes a couple unnecessary printlns in the test, and makes sure the test cleans up after itself.

          Show
          Sandy Ryza added a comment - Uploaded a patch that fixes the typo, removes a couple unnecessary printlns in the test, and makes sure the test cleans up after itself.
          Alejandro Abdelnur made changes -
          Status Reopened [ 4 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12566864/MAPREDUCE-2264-trunk-5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3287//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3287//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3287//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566864/MAPREDUCE-2264-trunk-5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3287//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3287//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3287//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          Sandy, please check the equals()/hash() in the CompressAwarePath class, findbugs is complaining because only one is implemented. By quickly looking at Path I don't think we need to override any of them. Would you please verify?

          Show
          Alejandro Abdelnur added a comment - Sandy, please check the equals()/hash() in the CompressAwarePath class, findbugs is complaining because only one is implemented. By quickly looking at Path I don't think we need to override any of them. Would you please verify?
          Sandy Ryza made changes -
          Attachment MAPREDUCE-2264-trunk-5.patch [ 12566880 ]
          Hide
          Sandy Ryza added a comment -

          equals() is overridden because findbugs complains if it is not (the rationale is here: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3284//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html). I uploaded a patch that adds a hashCode method as well.

          Show
          Sandy Ryza added a comment - equals() is overridden because findbugs complains if it is not (the rationale is here: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3284//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html ). I uploaded a patch that adds a hashCode method as well.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12566880/MAPREDUCE-2264-trunk-5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3288//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3288//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566880/MAPREDUCE-2264-trunk-5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3288//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3288//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          +1, lgtm. I'll commit tomorrow morning PST unless I see comments otherwise.

          Show
          Alejandro Abdelnur added a comment - +1, lgtm. I'll commit tomorrow morning PST unless I see comments otherwise.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #111 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/111/)
          Revering MAPREDUCE-2264 (Revision 1439561)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1439561
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #111 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/111/ ) Revering MAPREDUCE-2264 (Revision 1439561) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1439561 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1300 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1300/)
          Revering MAPREDUCE-2264 (Revision 1439561)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1439561
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1300 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1300/ ) Revering MAPREDUCE-2264 (Revision 1439561) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1439561 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1328 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1328/)
          Revering MAPREDUCE-2264 (Revision 1439561)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1439561
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1328 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1328/ ) Revering MAPREDUCE-2264 (Revision 1439561) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1439561 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMerger.java
          Hide
          Arun C Murthy added a comment -

          Sandy Ryza Are you updating patch for branch-1 too?

          Show
          Arun C Murthy added a comment - Sandy Ryza Are you updating patch for branch-1 too?
          Hide
          Alejandro Abdelnur added a comment -

          Arun, the patch for branch-1 is OK as it the CompressionAware class is already an extension of FileStatus. The prob is only in trunk/branch-2 were CompressionAware class was not extending Path.

          Show
          Alejandro Abdelnur added a comment - Arun, the patch for branch-1 is OK as it the CompressionAware class is already an extension of FileStatus. The prob is only in trunk/branch-2 were CompressionAware class was not extending Path.
          Hide
          Alejandro Abdelnur added a comment -

          Thanks (again) Sandy. Committed to trunk and branch-2. Already in branch-1 (we didn't need to revert it there).

          Show
          Alejandro Abdelnur added a comment - Thanks (again) Sandy. Committed to trunk and branch-2. Already in branch-1 (we didn't need to revert it there).
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3294 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3294/)
          MAPREDUCE-2264. Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1440076)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440076
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMerger.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3294 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3294/ ) MAPREDUCE-2264 . Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1440076) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440076 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMerger.java
          Hide
          Arun C Murthy added a comment -

          Alejandro - It would have been nicer to wait for my review since it was in progress, mind not doing it again? If I have comments I'll re-open this. FYI.

          Show
          Arun C Murthy added a comment - Alejandro - It would have been nicer to wait for my review since it was in progress, mind not doing it again? If I have comments I'll re-open this. FYI.
          Hide
          Alejandro Abdelnur added a comment -

          Arun, ??? I've asked yesterday evening, https://issues.apache.org/jira/browse/MAPREDUCE-2264?focusedCommentId=13565037&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13565037 and you commented on the patch after my comment asking, so I'd assumed you read it. Fell free to re-open if you find any issue.

          Show
          Alejandro Abdelnur added a comment - Arun, ??? I've asked yesterday evening, https://issues.apache.org/jira/browse/MAPREDUCE-2264?focusedCommentId=13565037&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13565037 and you commented on the patch after my comment asking, so I'd assumed you read it. Fell free to re-open if you find any issue.
          Hide
          Arun C Murthy added a comment -

          Comments:

          1. We shouldn't introduce a whole set of ctors if we can avoid it - particularly since Segment is just internal api. Proliferation of ctors is something to avoid.
          2. The Segment.getRawDataLength implementation is odd - won't it be better to just set rawLength = getLength() as default-value unless it's passed in explicitly?
          3. We seem to be doing localFS.getFileStatus(file).getLen() on CompressAwarePaths, won't it be easier to set it up so that we just call Segment.getLength() by, maybe, doing localFS.getFileStatus(file).getLen() in the Segment ctor or lazily in Segment.getLength?
          4. A coding nit: we should add a CompressAwarePath.create static method and use it rather than try to 'new' one in several places. Obviously a factory is nicer for unit-testing etc., but it's an overkill for now?
          Show
          Arun C Murthy added a comment - Comments: We shouldn't introduce a whole set of ctors if we can avoid it - particularly since Segment is just internal api. Proliferation of ctors is something to avoid. The Segment.getRawDataLength implementation is odd - won't it be better to just set rawLength = getLength() as default-value unless it's passed in explicitly? We seem to be doing localFS.getFileStatus(file).getLen() on CompressAwarePaths, won't it be easier to set it up so that we just call Segment.getLength() by, maybe, doing localFS.getFileStatus(file).getLen() in the Segment ctor or lazily in Segment.getLength? A coding nit: we should add a CompressAwarePath.create static method and use it rather than try to 'new' one in several places. Obviously a factory is nicer for unit-testing etc., but it's an overkill for now?
          Hide
          Arun C Murthy added a comment -

          I can see the confusion, but I'd appreciate if you could wait for a +1. Thanks.

          Show
          Arun C Murthy added a comment - I can see the confusion, but I'd appreciate if you could wait for a +1. Thanks.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #112 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/112/)
          MAPREDUCE-2264. Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1440076)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440076
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMerger.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #112 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/112/ ) MAPREDUCE-2264 . Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1440076) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440076 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMerger.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1301 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1301/)
          MAPREDUCE-2264. Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1440076)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440076
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMerger.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1301 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1301/ ) MAPREDUCE-2264 . Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1440076) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440076 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMerger.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1329 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1329/)
          MAPREDUCE-2264. Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1440076)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440076
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMerger.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1329 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1329/ ) MAPREDUCE-2264 . Job status exceeds 100% in some cases. (devaraj.k and sandyr via tucu) (Revision 1440076) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440076 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Merger.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMerger.java
          Hide
          Sandy Ryza added a comment -

          Thanks for the comments, Arun.

          1. Agreed that Segment has too many constructors. While the patch adds 2, there were 5 already, so the existing pattern was just being followed. What would be the best way to fix this? There's also some confusing formatting in some of the old ones that could be improved.
          2. Agreed.
          3. From what I can tell, this is what is already done the Segment constructor. localFS.getFileStatus(file).getLen() is called again/separately on the path to calculate the onDiskBytes. We could save a call to getFileStatus constructing the Segment first and calculating the onDiskBytes after. (For reference, I am looking at finalMerge). This issue existed before the patch as well.
          4. What is the advantage of a create() method over a constructor?

          With the patch working in its current incarnation, would it be better to file a new JIRA for the cleanups or do it in this one?

          Show
          Sandy Ryza added a comment - Thanks for the comments, Arun. 1. Agreed that Segment has too many constructors. While the patch adds 2, there were 5 already, so the existing pattern was just being followed. What would be the best way to fix this? There's also some confusing formatting in some of the old ones that could be improved. 2. Agreed. 3. From what I can tell, this is what is already done the Segment constructor. localFS.getFileStatus(file).getLen() is called again/separately on the path to calculate the onDiskBytes. We could save a call to getFileStatus constructing the Segment first and calculating the onDiskBytes after. (For reference, I am looking at finalMerge). This issue existed before the patch as well. 4. What is the advantage of a create() method over a constructor? With the patch working in its current incarnation, would it be better to file a new JIRA for the cleanups or do it in this one?
          Hide
          Sandy Ryza added a comment -

          Arun,

          Haven't heard from you on this. As most of the changes are unrelated to the original issue, I'll mark this as resolved and work on a cleanup JIRA tomorrow unless you say otherwise?

          Show
          Sandy Ryza added a comment - Arun, Haven't heard from you on this. As most of the changes are unrelated to the original issue, I'll mark this as resolved and work on a cleanup JIRA tomorrow unless you say otherwise?
          Hide
          Arun C Murthy added a comment -

          If you can attach an addendum patch asap I'd be thrilled... thanks.

          Show
          Arun C Murthy added a comment - If you can attach an addendum patch asap I'd be thrilled... thanks.
          Sandy Ryza made changes -
          Attachment MAPREDUCE-2264-trunk-addendum.patch [ 12568148 ]
          Hide
          Sandy Ryza added a comment -

          Attached addendum patch

          Show
          Sandy Ryza added a comment - Attached addendum patch
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Mariappan Asokan added a comment -

          The patch has introduced a class cast exception. Please see MAPREDUCE-5034

          Show
          Mariappan Asokan added a comment - The patch has introduced a class cast exception. Please see MAPREDUCE-5034
          Hide
          Sandy Ryza added a comment -

          This looks the same as the issue that was reported by Chris on January 27th - is it possible you're using the version of the patch that was reverted? The way to check would be to see whether CompressAwarePath extends Path.

          Show
          Sandy Ryza added a comment - This looks the same as the issue that was reported by Chris on January 27th - is it possible you're using the version of the patch that was reverted? The way to check would be to see whether CompressAwarePath extends Path.

            People

            • Assignee:
              Devaraj K
              Reporter:
              Adam Kramer
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development