Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1705

Archiving and Purging of parity files should handle globbed policies

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Archiving (har) and purging of parity files don't work in policies whose source is a globbed path.

        Activity

        Hide
        dhruba borthakur added a comment -

        I just committed this. Thanks Rodrigo!

        Show
        dhruba borthakur added a comment - I just committed this. Thanks Rodrigo!
        Hide
        Rodrigo Schmidt added a comment -

        Great! Thanks!

        Show
        Rodrigo Schmidt added a comment - Great! Thanks!
        Hide
        dhruba borthakur added a comment -

        The two failed tests are org.apache.hadoop.mapred.TestBadRecords.testBadMapRed org.apache.hadoop.mapreduce.TestJobACLs.testACLS asnd is not related to RAID.

        Show
        dhruba borthakur added a comment - The two failed tests are org.apache.hadoop.mapred.TestBadRecords.testBadMapRed org.apache.hadoop.mapreduce.TestJobACLs.testACLS asnd is not related to RAID.
        Hide
        dhruba borthakur added a comment -

        I agree, i reran the tests now and it works as you explained. I will commit this patch.

        Show
        dhruba borthakur added a comment - I agree, i reran the tests now and it works as you explained. I will commit this patch.
        Hide
        Rodrigo Schmidt added a comment -

        Dhruba, I just double checked. The new unit tests (TestRaidPurge and TestRaidHar) block if the patch is not applied to RaidNode.java.

        They pass if they are not changed, and they also pass if the whole patch is applied. Maybe you didn't apply the changes to the unit tests, or you applied the whole patch when you tested.

        As for the Hadoop QA output, the two core tests that failed are unrelated to Raid.

        Show
        Rodrigo Schmidt added a comment - Dhruba, I just double checked. The new unit tests (TestRaidPurge and TestRaidHar) block if the patch is not applied to RaidNode.java. They pass if they are not changed, and they also pass if the whole patch is applied. Maybe you didn't apply the changes to the unit tests, or you applied the whole patch when you tested. As for the Hadoop QA output, the two core tests that failed are unrelated to Raid.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12442833/MAPREDUCE-1705.0.patch
        against trunk revision 937924.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/139/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/139/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/139/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/139/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442833/MAPREDUCE-1705.0.patch against trunk revision 937924. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/139/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/139/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/139/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/139/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        Code looks good. I tried running the test case without the patch applied and the test case succeeded. Let me re-try.

        Show
        dhruba borthakur added a comment - Code looks good. I tried running the test case without the patch applied and the test case succeeded. Let me re-try.
        Hide
        Rodrigo Schmidt added a comment -

        First patch.

        Show
        Rodrigo Schmidt added a comment - First patch.

          People

          • Assignee:
            Rodrigo Schmidt
            Reporter:
            Rodrigo Schmidt
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development