Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1512

RAID could use HarFileSystem directly instead of FileSystem.get

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Makes the code run slightly faster and avoids possible problems in matching the right filesystem like the stale cache reported in HADOOP-6097.

      This is a minor improvement for trunk, but it is really helpful for people running RAID on earlier releases susceptible to HADOOP-6097, since RAID would crash on them.

      1. MAPREDUCE-1512.patch
        1 kB
        Rodrigo Schmidt
      2. MAPREDUCE-1512.1.patch
        9 kB
        Rodrigo Schmidt

        Issue Links

          Activity

          Hide
          dhruba borthakur added a comment -

          +1

          Show
          dhruba borthakur added a comment - +1
          Hide
          Rodrigo Schmidt added a comment -

          Please don't commit this patch yet. I'll take advantage of this JIRA to do some refactoring of the code that finds parity files. I'll submit a new patch soon.

          Show
          Rodrigo Schmidt added a comment - Please don't commit this patch yet. I'll take advantage of this JIRA to do some refactoring of the code that finds parity files. I'll submit a new patch soon.
          Hide
          Rodrigo Schmidt added a comment -

          I'm adding a dependency because these issues happen on the same source file, and I would like to tackle them one at a time.

          Show
          Rodrigo Schmidt added a comment - I'm adding a dependency because these issues happen on the same source file, and I would like to tackle them one at a time.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12436295/MAPREDUCE-1512.patch
          against trunk revision 911519.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436295/MAPREDUCE-1512.patch against trunk revision 911519. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/console This message is automatically generated.
          Hide
          Rodrigo Schmidt added a comment -

          This is a simple improvement, but I'm taking advantage of this JIRA to do some code refactoring to improve the structure of the RaidNode.

          Show
          Rodrigo Schmidt added a comment - This is a simple improvement, but I'm taking advantage of this JIRA to do some code refactoring to improve the structure of the RaidNode.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437733/MAPREDUCE-1512.1.patch
          against trunk revision 918864.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437733/MAPREDUCE-1512.1.patch against trunk revision 918864. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/console This message is automatically generated.
          Hide
          Rodrigo Schmidt added a comment -

          This patch is mostly refactoring the code. It simplifies some things, and optimizes others. I don't think we need new tests since there is no bug or new feature associated with it.

          Show
          Rodrigo Schmidt added a comment - This patch is mostly refactoring the code. It simplifies some things, and optimizes others. I don't think we need new tests since there is no bug or new feature associated with it.
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Rodrigo!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Rodrigo!
          Hide
          Rodrigo Schmidt added a comment -

          Thanks, Dhruba!

          Now I'll submit the patch for MAPREDUCE-1518.

          Cheers,
          Rodrigo

          Show
          Rodrigo Schmidt added a comment - Thanks, Dhruba! Now I'll submit the patch for MAPREDUCE-1518 . Cheers, Rodrigo
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #260 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/260/)
          . RAID uses HarFileSystem directly instead of
          FileSystem.get (Rodrigo Schmidt via dhruba)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #260 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/260/ ) . RAID uses HarFileSystem directly instead of FileSystem.get (Rodrigo Schmidt via dhruba)
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #250 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/250/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #250 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/250/ )

            People

            • Assignee:
              Rodrigo Schmidt
              Reporter:
              Rodrigo Schmidt
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development