Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1091

Implement listStatus that returns an Iterator of FileStatus

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This jira is for an efficient implementation of the method Iterator<FileStatus> listStatus(Path) in HDFS.

      1. HdfsListStatusIterator1.patch
        8 kB
        Hairong Kuang
      2. HdfsListStatusIterator.patch
        8 kB
        Hairong Kuang

        Issue Links

          Activity

          Hide
          Hairong Kuang added a comment -

          Here is a patch that provides an efficient implementation of listStatus that returns an iterator in Hdfs. It modifies TestFileStatus to use the new FileContext#listStatus API implemented in HADOOP-6692. It also included a minor fix on Hdfs#listStatus that is supposed to get into HDFS-985 but did not make it by mistake.

          Show
          Hairong Kuang added a comment - Here is a patch that provides an efficient implementation of listStatus that returns an iterator in Hdfs. It modifies TestFileStatus to use the new FileContext#listStatus API implemented in HADOOP-6692 . It also included a minor fix on Hdfs#listStatus that is supposed to get into HDFS-985 but did not make it by mistake.
          Hide
          Konstantin Shvachko added a comment -
          1. @Override notation for Hdfs.listStatusIterator() should be un-commented. It would be also useful to add a comment what it overrides, like
            @Override // AbstractFileSystem
            
          2. In the Iterator<FileStatus> returned by Hdfs.listStatusIterator() the partialListing member seem to be redundant. It is already a member of DirectoryListing, so you can use thisListing.getPartialListing() instead.
          3. It would be good to have an explanatory message when you throw NoSuchElementException() and UnsupportedOperationException().
          Show
          Konstantin Shvachko added a comment - @Override notation for Hdfs.listStatusIterator() should be un-commented. It would be also useful to add a comment what it overrides, like @Override // AbstractFileSystem In the Iterator<FileStatus> returned by Hdfs.listStatusIterator() the partialListing member seem to be redundant. It is already a member of DirectoryListing , so you can use thisListing.getPartialListing() instead. It would be good to have an explanatory message when you throw NoSuchElementException() and UnsupportedOperationException() .
          Hide
          Hairong Kuang added a comment -

          Thank Konstantin for his review comments. HdfsListStatusItor1.patch addresses his comments.

          Show
          Hairong Kuang added a comment - Thank Konstantin for his review comments. HdfsListStatusItor1.patch addresses his comments.
          Hide
          Konstantin Shvachko added a comment -

          +1 This looks good.

          Show
          Konstantin Shvachko added a comment - +1 This looks good.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12443008/HdfsListStatusIterator1.patch
          against trunk revision 937914.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/329/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/329/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/329/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/329/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12443008/HdfsListStatusIterator1.patch against trunk revision 937914. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/329/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/329/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/329/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/329/console This message is automatically generated.
          Hide
          Hairong Kuang added a comment -

          Not sure why contrib tests did not pass. The console does not have any log information of the failed test. The failed test passed on both my Mac machine and Linux box. I do not think my patch touches the failed test. I will commit the patch for now. If we find any problem with it, I will revert it immediately.

          Show
          Hairong Kuang added a comment - Not sure why contrib tests did not pass. The console does not have any log information of the failed test. The failed test passed on both my Mac machine and Linux box. I do not think my patch touches the failed test. I will commit the patch for now. If we find any problem with it, I will revert it immediately.

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development