Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1512

RAID could use HarFileSystem directly instead of FileSystem.get

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Makes the code run slightly faster and avoids possible problems in matching the right filesystem like the stale cache reported in HADOOP-6097.

      This is a minor improvement for trunk, but it is really helpful for people running RAID on earlier releases susceptible to HADOOP-6097, since RAID would crash on them.

      1. MAPREDUCE-1512.1.patch
        9 kB
        Rodrigo Schmidt
      2. MAPREDUCE-1512.patch
        1 kB
        Rodrigo Schmidt

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          2d 15h 57m 1 Rodrigo Schmidt 22/Feb/10 00:07
          Open Open Patch Available Patch Available
          9d 12h 23m 2 Rodrigo Schmidt 03/Mar/10 12:25
          Patch Available Patch Available Resolved Resolved
          1d 8h 25m 1 dhruba borthakur 04/Mar/10 20:51
          Resolved Resolved Closed Closed
          173d 29m 1 Tom White 24/Aug/10 22:20
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Tom White made changes -
          Fix Version/s 0.22.0 [ 12314184 ]
          Fix Version/s 0.21.0 [ 12314045 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #250 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/250/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #250 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/250/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #260 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/260/)
          . RAID uses HarFileSystem directly instead of
          FileSystem.get (Rodrigo Schmidt via dhruba)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #260 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/260/ ) . RAID uses HarFileSystem directly instead of FileSystem.get (Rodrigo Schmidt via dhruba)
          Hide
          Rodrigo Schmidt added a comment -

          Thanks, Dhruba!

          Now I'll submit the patch for MAPREDUCE-1518.

          Cheers,
          Rodrigo

          Show
          Rodrigo Schmidt added a comment - Thanks, Dhruba! Now I'll submit the patch for MAPREDUCE-1518 . Cheers, Rodrigo
          dhruba borthakur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.22.0 [ 12314184 ]
          Resolution Fixed [ 1 ]
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Rodrigo!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Rodrigo!
          Hide
          Rodrigo Schmidt added a comment -

          This patch is mostly refactoring the code. It simplifies some things, and optimizes others. I don't think we need new tests since there is no bug or new feature associated with it.

          Show
          Rodrigo Schmidt added a comment - This patch is mostly refactoring the code. It simplifies some things, and optimizes others. I don't think we need new tests since there is no bug or new feature associated with it.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437733/MAPREDUCE-1512.1.patch
          against trunk revision 918864.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437733/MAPREDUCE-1512.1.patch against trunk revision 918864. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/499/console This message is automatically generated.
          Rodrigo Schmidt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Rodrigo Schmidt made changes -
          Attachment MAPREDUCE-1512.1.patch [ 12437733 ]
          Hide
          Rodrigo Schmidt added a comment -

          This is a simple improvement, but I'm taking advantage of this JIRA to do some code refactoring to improve the structure of the RaidNode.

          Show
          Rodrigo Schmidt added a comment - This is a simple improvement, but I'm taking advantage of this JIRA to do some code refactoring to improve the structure of the RaidNode.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12436295/MAPREDUCE-1512.patch
          against trunk revision 911519.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436295/MAPREDUCE-1512.patch against trunk revision 911519. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/330/console This message is automatically generated.
          Rodrigo Schmidt made changes -
          Link This issue is depended upon by MAPREDUCE-1518 [ MAPREDUCE-1518 ]
          Hide
          Rodrigo Schmidt added a comment -

          I'm adding a dependency because these issues happen on the same source file, and I would like to tackle them one at a time.

          Show
          Rodrigo Schmidt added a comment - I'm adding a dependency because these issues happen on the same source file, and I would like to tackle them one at a time.
          Rodrigo Schmidt made changes -
          Link This issue depends upon MAPREDUCE-1510 [ MAPREDUCE-1510 ]
          Rodrigo Schmidt made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Rodrigo Schmidt added a comment -

          Please don't commit this patch yet. I'll take advantage of this JIRA to do some refactoring of the code that finds parity files. I'll submit a new patch soon.

          Show
          Rodrigo Schmidt added a comment - Please don't commit this patch yet. I'll take advantage of this JIRA to do some refactoring of the code that finds parity files. I'll submit a new patch soon.
          Hide
          dhruba borthakur added a comment -

          +1

          Show
          dhruba borthakur added a comment - +1
          Rodrigo Schmidt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Rodrigo Schmidt [ rschmidt ]
          Rodrigo Schmidt made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-1512.patch [ 12436295 ]
          Rodrigo Schmidt created issue -

            People

            • Assignee:
              Rodrigo Schmidt
              Reporter:
              Rodrigo Schmidt
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development