Hadoop Common
  1. Hadoop Common
  2. HADOOP-2027

FileSystem should provide byte ranges for file locations

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.17.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      New FileSystem API getFileBlockLocations to return the number of bytes in each block in a file via a single rpc to the namenode to speed up job planning. Deprecates getFileCacheHints.

      Description

      FileSystem's getFileCacheHints should be replaced with something more useful. I'd suggest replacing getFileCacheHints with a new method:

      BlockLocation[] getFileLocations(Path file, long offset, long range) throws IOException;
      

      and adding

      class BlockLocation implements Writable {
        String[] getHosts();
        long getOffset();
        long getLength();
      }
      
      1. HADOOP-2027-1.patch
        22 kB
        Lohit Vijayarenu
      2. HADOOP-2027-10.patch
        35 kB
        Lohit Vijayarenu
      3. HADOOP-2027-14.patch
        38 kB
        Lohit Vijayarenu
      4. HADOOP-2027-2.patch
        33 kB
        Lohit Vijayarenu
      5. HADOOP-2027-3.patch
        33 kB
        Lohit Vijayarenu
      6. HADOOP-2027-4.patch
        29 kB
        Lohit Vijayarenu
      7. HADOOP-2027-5.patch
        31 kB
        Lohit Vijayarenu
      8. HADOOP-2027-6.patch
        35 kB
        Lohit Vijayarenu
      9. HADOOP-2027-7.patch
        31 kB
        Lohit Vijayarenu
      10. HADOOP-2027-8.patch
        35 kB
        Lohit Vijayarenu
      11. HADOOP-2027-9.patch
        35 kB
        Lohit Vijayarenu
      12. HADOOP-2559-11.patch
        37 kB
        Lohit Vijayarenu
      13. HADOOP-2559-12.patch
        39 kB
        Lohit Vijayarenu
      14. HADOOP-2559-13.patch
        38 kB
        Lohit Vijayarenu

        Issue Links

          Activity

          Hide
          Owen O'Malley added a comment -

          Note that we also need Map/Reduce to use the new method so that it only does one call per a file to get block sizes. This would require that FileSplit have a new constructor that takes an array of locations rather than computing it on demand. The locations do NOT need to be serialized in the read/write fields methods. FileInputFormat should use a single call to getFileLocations rather than the current getSize, getBlockSize, and getFileCacheHints (down in FileSplit).

          Show
          Owen O'Malley added a comment - Note that we also need Map/Reduce to use the new method so that it only does one call per a file to get block sizes. This would require that FileSplit have a new constructor that takes an array of locations rather than computing it on demand. The locations do NOT need to be serialized in the read/write fields methods. FileInputFormat should use a single call to getFileLocations rather than the current getSize, getBlockSize, and getFileCacheHints (down in FileSplit).
          Hide
          Lohit Vijayarenu added a comment -

          Thanks Owen. Attached patch includes
          1. new API getFileBlockLocations which invokes getBlockLocations to return BlockLocation[]
          2. Changes FileSplit to store host information and return when getLocations() is invoked
          3. Change FileInputFormat to one call of getFileBlockLocations and store host information in FileSplit using new constructor

          I ran the unit test and do not see failures. Will test benchmark and report the timings.

          Show
          Lohit Vijayarenu added a comment - Thanks Owen. Attached patch includes 1. new API getFileBlockLocations which invokes getBlockLocations to return BlockLocation[] 2. Changes FileSplit to store host information and return when getLocations() is invoked 3. Change FileInputFormat to one call of getFileBlockLocations and store host information in FileSplit using new constructor I ran the unit test and do not see failures. Will test benchmark and report the timings.
          Hide
          Lohit Vijayarenu added a comment -

          I ran sort (twice) on 100 nodes on trunk+this patch. It took 28.4 and 27.3 minutes. Mukund mentioned it took 29.04 min on trunk.

          Show
          Lohit Vijayarenu added a comment - I ran sort (twice) on 100 nodes on trunk+this patch. It took 28.4 and 27.3 minutes. Mukund mentioned it took 29.04 min on trunk.
          Hide
          Lohit Vijayarenu added a comment -

          Making this PA

          Show
          Lohit Vijayarenu added a comment - Making this PA
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12375144/HADOOP-2027-1.patch
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 21 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac -1. The applied patch generated 619 javac compiler warnings (more than the trunk's current 608 warnings).

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs -1. The patch appears to introduce 3 new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1781/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1781/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1781/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1781/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12375144/HADOOP-2027-1.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 21 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac -1. The applied patch generated 619 javac compiler warnings (more than the trunk's current 608 warnings). release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to introduce 3 new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1781/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1781/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1781/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1781/console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          You should deprecate the old FileSplit constructor and make it call the new one.

          Set hosts to null in readFields.

          Show
          Owen O'Malley added a comment - You should deprecate the old FileSplit constructor and make it call the new one. Set hosts to null in readFields.
          Hide
          Owen O'Malley added a comment -

          You should:
          1. Not use strings of '*' around your javadoc.
          2. Fill in the javadoc of public methods in BlockLocation.
          3. I'd prefer using String[] in BlockLocation, since the API uses String rather than Text.
          4. FileSystem.getFileBlockLocations should just pass the desired values into the constructor rather than setting them all, same for DFSClient
          5. The indentation in FileInputFormat should bring lines to the open of the paren
          6. Fix the calls to the now deprecated methods.

          Thanks! I'm looking forward to this patch.

          Show
          Owen O'Malley added a comment - You should: 1. Not use strings of '*' around your javadoc. 2. Fill in the javadoc of public methods in BlockLocation. 3. I'd prefer using String[] in BlockLocation, since the API uses String rather than Text. 4. FileSystem.getFileBlockLocations should just pass the desired values into the constructor rather than setting them all, same for DFSClient 5. The indentation in FileInputFormat should bring lines to the open of the paren 6. Fix the calls to the now deprecated methods. Thanks! I'm looking forward to this patch.
          Hide
          Lohit Vijayarenu added a comment -

          Incorporating changes suggested by Owen, removed javac warnings which were due to deprecated calls. I could not get rid of 2 of them which were deprecated calls to Kosmos FileSystem. I am submitting this patch for QA run. Will either try to get those fixed or open new JIRA to fix it

          Show
          Lohit Vijayarenu added a comment - Incorporating changes suggested by Owen, removed javac warnings which were due to deprecated calls. I could not get rid of 2 of them which were deprecated calls to Kosmos FileSystem. I am submitting this patch for QA run. Will either try to get those fixed or open new JIRA to fix it
          Hide
          Lohit Vijayarenu added a comment -

          Resubmitting against latest trunk

          Show
          Lohit Vijayarenu added a comment - Resubmitting against latest trunk
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12375464/HADOOP-2027-2.patch
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 30 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac -1. The applied patch generated 608 javac compiler warnings (more than the trunk's current 604 warnings).

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs -1. The patch appears to introduce 3 new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1788/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1788/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1788/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1788/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12375464/HADOOP-2027-2.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 30 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac -1. The applied patch generated 608 javac compiler warnings (more than the trunk's current 604 warnings). release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to introduce 3 new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1788/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1788/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1788/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1788/console This message is automatically generated.
          Hide
          Lohit Vijayarenu added a comment -

          Another try fixing findbugs

          Show
          Lohit Vijayarenu added a comment - Another try fixing findbugs
          Hide
          Lohit Vijayarenu added a comment -

          I deprecated KFS getFileCacheHints and modified KFSEmulationImpl to call local Filesytems getFileBlockLocations
          There are few javac warnings, which are due to other deprecated APIs like listPaths globPaths. I am attaching this patch against trunk

          Show
          Lohit Vijayarenu added a comment - I deprecated KFS getFileCacheHints and modified KFSEmulationImpl to call local Filesytems getFileBlockLocations There are few javac warnings, which are due to other deprecated APIs like listPaths globPaths. I am attaching this patch against trunk
          Hide
          Lohit Vijayarenu added a comment -

          Canceling and resubmitting patch

          Show
          Lohit Vijayarenu added a comment - Canceling and resubmitting patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12375522/HADOOP-2027-5.patch
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 33 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs -1. The patch appears to cause Findbugs to fail.

          core tests -1. The patch failed core unit tests.

          contrib tests -1. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1790/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1790/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1790/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12375522/HADOOP-2027-5.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 33 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to cause Findbugs to fail. core tests -1. The patch failed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1790/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1790/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1790/console This message is automatically generated.
          Hide
          Lohit Vijayarenu added a comment -

          I missed BlockLocations.java (new file) so build failed. I will resubmit it

          Show
          Lohit Vijayarenu added a comment - I missed BlockLocations.java (new file) so build failed. I will resubmit it
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12375528/HADOOP-2027-6.patch
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 33 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac -1. The applied patch generated 605 javac compiler warnings (more than the trunk's current 603 warnings).

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1791/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1791/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1791/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1791/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12375528/HADOOP-2027-6.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 33 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac -1. The applied patch generated 605 javac compiler warnings (more than the trunk's current 603 warnings). release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1791/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1791/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1791/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1791/console This message is automatically generated.
          Hide
          Lohit Vijayarenu added a comment -

          The javac warnings were expected due to other deprecated APIs.

          Show
          Lohit Vijayarenu added a comment - The javac warnings were expected due to other deprecated APIs.
          Hide
          Lohit Vijayarenu added a comment -

          Found the 2 additional warnings, they were from PhasedFileSystem.java
          > [javac] /zonestorage/hudson/home/hudson/hudson/jobs/Hadoop-Patch/workspace/trunk/src/java/org/apache/hadoop/mapred/PhasedFileSystem.java:300: warning: [deprecation] getFileCacheHints(org.apache.hadoop.fs.Path,long,long) in org.apache.hadoop.fs.FilterFileSystem has been deprecated> [javac] /zonestorage/hudson/home/hudson/hudson/jobs/Hadoop-Patch/workspace/trunk/src/java/org/apache/hadoop/mapred/PhasedFileSystem.java:300: warning: [deprecation] getFileCacheHints(org.apache.hadoop.fs.Path,long,long) in org.apache.hadoop.fs.FileSystem has been deprecated

          Show
          Lohit Vijayarenu added a comment - Found the 2 additional warnings, they were from PhasedFileSystem.java > [javac] /zonestorage/hudson/home/hudson/hudson/jobs/Hadoop-Patch/workspace/trunk/src/java/org/apache/hadoop/mapred/PhasedFileSystem.java:300: warning: [deprecation] getFileCacheHints(org.apache.hadoop.fs.Path,long,long) in org.apache.hadoop.fs.FilterFileSystem has been deprecated> [javac] /zonestorage/hudson/home/hudson/hudson/jobs/Hadoop-Patch/workspace/trunk/src/java/org/apache/hadoop/mapred/PhasedFileSystem.java:300: warning: [deprecation] getFileCacheHints(org.apache.hadoop.fs.Path,long,long) in org.apache.hadoop.fs.FileSystem has been deprecated
          Hide
          Lohit Vijayarenu added a comment -

          Attaching same patch by regenerating against trunk.

          Show
          Lohit Vijayarenu added a comment - Attaching same patch by regenerating against trunk.
          Hide
          Lohit Vijayarenu added a comment -

          sorry had missed BlockLocation again

          Show
          Lohit Vijayarenu added a comment - sorry had missed BlockLocation again
          Hide
          Lohit Vijayarenu added a comment -

          Uploading new one with comments from dhruba.

          Show
          Lohit Vijayarenu added a comment - Uploading new one with comments from dhruba.
          Hide
          Lohit Vijayarenu added a comment -

          Dhurba suggested it would be good to have information about host:port which is already provided by namenode call.
          So i have one more API getNames() which is similar to DatanodeID's getName, this returns hostname:port and getHosts() returns hostnames as earlier. He suggest this is useful when we consider running 2 datanodes on same node. Attached is the patch which address this.

          Show
          Lohit Vijayarenu added a comment - Dhurba suggested it would be good to have information about host:port which is already provided by namenode call. So i have one more API getNames() which is similar to DatanodeID's getName, this returns hostname:port and getHosts() returns hostnames as earlier. He suggest this is useful when we consider running 2 datanodes on same node. Attached is the patch which address this.
          Hide
          Lohit Vijayarenu added a comment -

          making this PA

          Show
          Lohit Vijayarenu added a comment - making this PA
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12376764/HADOOP-2027-9.patch
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 33 new or modified tests.

          patch -1. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1866/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12376764/HADOOP-2027-9.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 33 new or modified tests. patch -1. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1866/console This message is automatically generated.
          Hide
          Lohit Vijayarenu added a comment -

          Regenerating against trunk. Tested this patch, applies clean on trunk.

          Show
          Lohit Vijayarenu added a comment - Regenerating against trunk. Tested this patch, applies clean on trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12376793/HADOOP-2027-10.patch
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 33 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac -1. The applied patch generated 617 javac compiler warnings (more than the trunk's current 614 warnings).

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests -1. The patch failed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1874/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1874/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1874/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1874/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12376793/HADOOP-2027-10.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 33 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac -1. The applied patch generated 617 javac compiler warnings (more than the trunk's current 614 warnings). release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1874/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1874/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1874/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1874/console This message is automatically generated.
          Hide
          Lohit Vijayarenu added a comment -

          TestRackAwareTaskPlacement was failing because of my latest changes. While adding getNames() method, I tried to derive getHosts() from string returned by getName(), this had ipaddress:port. But we needed hostnames. So, I created 2 separate arrays within BlockLocation. Now both getHosts() and getNames() return the expected output. Attaching a new patch.

          Show
          Lohit Vijayarenu added a comment - TestRackAwareTaskPlacement was failing because of my latest changes. While adding getNames() method, I tried to derive getHosts() from string returned by getName(), this had ipaddress:port. But we needed hostnames. So, I created 2 separate arrays within BlockLocation. Now both getHosts() and getNames() return the expected output. Attaching a new patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12377229/HADOOP-2559-11.patch
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 36 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac -1. The applied patch generated 593 javac compiler warnings (more than the trunk's current 590 warnings).

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests -1. The patch failed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1902/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1902/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1902/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1902/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12377229/HADOOP-2559-11.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 36 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac -1. The applied patch generated 593 javac compiler warnings (more than the trunk's current 590 warnings). release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1902/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1902/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1902/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1902/console This message is automatically generated.
          Hide
          Lohit Vijayarenu added a comment -

          modified TestTextInputFormat. With the updated patch, if file is of 0 length it will not be added to the splits.

          Show
          Lohit Vijayarenu added a comment - modified TestTextInputFormat. With the updated patch, if file is of 0 length it will not be added to the splits.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12377232/HADOOP-2559-12.patch
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 42 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac -1. The applied patch generated 592 javac compiler warnings (more than the trunk's current 590 warnings).

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1903/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1903/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1903/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1903/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12377232/HADOOP-2559-12.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 42 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac -1. The applied patch generated 592 javac compiler warnings (more than the trunk's current 590 warnings). release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1903/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1903/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1903/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1903/console This message is automatically generated.
          Hide
          Lohit Vijayarenu added a comment -

          All tests passed. 2 additional javac warnings are due to PhasedFileSystem.java as expected.

          Show
          Lohit Vijayarenu added a comment - All tests passed. 2 additional javac warnings are due to PhasedFileSystem.java as expected.
          Hide
          Lohit Vijayarenu added a comment -

          For now, creating empty hosts array when of input file is zero length. Owen opened HADOOP-2952 to address zero length files. Attaching another patch.

          Show
          Lohit Vijayarenu added a comment - For now, creating empty hosts array when of input file is zero length. Owen opened HADOOP-2952 to address zero length files. Attaching another patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12377271/HADOOP-2559-13.patch
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 39 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac -1. The applied patch generated 592 javac compiler warnings (more than the trunk's current 590 warnings).

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1905/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1905/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1905/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1905/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12377271/HADOOP-2559-13.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 39 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac -1. The applied patch generated 592 javac compiler warnings (more than the trunk's current 590 warnings). release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1905/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1905/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1905/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1905/console This message is automatically generated.
          Hide
          Lohit Vijayarenu added a comment -

          All tests passed. 2 additional javac warnings are due to PhasedFileSystem.java as expected.

          Show
          Lohit Vijayarenu added a comment - All tests passed. 2 additional javac warnings are due to PhasedFileSystem.java as expected.
          Hide
          Lohit Vijayarenu added a comment -

          Attaching another patch after changing getFileCacheHints in FileSystem.java and KFS.

          Show
          Lohit Vijayarenu added a comment - Attaching another patch after changing getFileCacheHints in FileSystem.java and KFS.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12377568/HADOOP-2027-14.patch
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 39 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac -1. The applied patch generated 599 javac compiler warnings (more than the trunk's current 598 warnings).

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1938/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1938/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1938/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1938/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12377568/HADOOP-2027-14.patch against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 39 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac -1. The applied patch generated 599 javac compiler warnings (more than the trunk's current 598 warnings). release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1938/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1938/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1938/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1938/console This message is automatically generated.
          Hide
          Lohit Vijayarenu added a comment -

          javac warning from PhasedFileSystem. All other tests passed.

          Show
          Lohit Vijayarenu added a comment - javac warning from PhasedFileSystem. All other tests passed.
          Hide
          Owen O'Malley added a comment -

          I just committed this. Thanks, Lohit!

          Show
          Owen O'Malley added a comment - I just committed this. Thanks, Lohit!
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #434 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/434/ )

            People

            • Assignee:
              Lohit Vijayarenu
              Reporter:
              Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development