Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14169

Implement listStatusIterator, listLocatedStatus for ViewFs

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.5
    • Fix Version/s: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
    • Component/s: viewfs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Similar to what HADOOP-11812 did for ViewFileSystem, currently ViewFs does not respect optimizations to listStatusIterator or listLocatedStatus, using the naive implementations within AbstractFileSystem. This can cause performance issues if iterating over a large directory, especially if the locations are also needed. This can be easily fixed.

      1. HADOOP-14169.002.patch
        12 kB
        Erik Krogen
      2. HADOOP-14169.001.patch
        12 kB
        Erik Krogen
      3. HADOOP-14169.000.patch
        12 kB
        Erik Krogen

        Issue Links

          Activity

          Hide
          xkrogen Erik Krogen added a comment -

          Attaching v0 patch with a very simple fix to implement listStatusIterator/listLocatedStatus on ChRootedFs and listStatusIterator on ViewFs. Minor refactor in ViewFs for code sharing between listStatusIterator and listLocatedStatus.

          Gera Shegalov, Chris Nauroth, pinging based on previous involvement with the similar work in HADOOP-11812 - are either of you interested in reviewing?

          Show
          xkrogen Erik Krogen added a comment - Attaching v0 patch with a very simple fix to implement listStatusIterator / listLocatedStatus on ChRootedFs and listStatusIterator on ViewFs . Minor refactor in ViewFs for code sharing between listStatusIterator and listLocatedStatus . Gera Shegalov , Chris Nauroth , pinging based on previous involvement with the similar work in HADOOP-11812 - are either of you interested in reviewing?
          Hide
          xkrogen Erik Krogen added a comment -

          v001 patch removes unused imports...

          Show
          xkrogen Erik Krogen added a comment - v001 patch removes unused imports...
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 16s trunk passed
          +1 compile 10m 15s trunk passed
          +1 checkstyle 0m 42s trunk passed
          +1 mvnsite 1m 7s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 1m 31s trunk passed
          +1 javadoc 0m 54s trunk passed
          +1 mvninstall 0m 36s the patch passed
          +1 compile 9m 52s the patch passed
          +1 javac 9m 52s the patch passed
          -0 checkstyle 0m 42s hadoop-common-project/hadoop-common: The patch generated 4 new + 164 unchanged - 7 fixed = 168 total (was 171)
          +1 mvnsite 1m 5s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 36s the patch passed
          +1 javadoc 0m 54s the patch passed
          -1 unit 8m 12s hadoop-common in the patch failed.
          +1 asflicense 0m 43s The patch does not generate ASF License warnings.
          53m 44s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14169
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857380/HADOOP-14169.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d5799293dc07 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9649c27
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11802/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11802/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11802/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11802/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 16s trunk passed +1 compile 10m 15s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 1m 31s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 9m 52s the patch passed +1 javac 9m 52s the patch passed -0 checkstyle 0m 42s hadoop-common-project/hadoop-common: The patch generated 4 new + 164 unchanged - 7 fixed = 168 total (was 171) +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 36s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 8m 12s hadoop-common in the patch failed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 53m 44s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14169 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857380/HADOOP-14169.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d5799293dc07 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9649c27 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11802/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11802/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11802/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11802/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          v002 patch addresses the few minor checkstyle issues

          Show
          xkrogen Erik Krogen added a comment - v002 patch addresses the few minor checkstyle issues
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Hi Erik Krogen, unless Chris Nauroth or someone else gets to it before me, I am happy to take a look too but probably not before in 2 weeks.

          Show
          jira.shegalov Gera Shegalov added a comment - Hi Erik Krogen , unless Chris Nauroth or someone else gets to it before me, I am happy to take a look too but probably not before in 2 weeks.
          Hide
          shv Konstantin Shvachko added a comment -

          This sounds like a good optimization for viewfs and a pretty clean patch.
          Will commit it soon, unless other people want to review it as well.
          +1

          Show
          shv Konstantin Shvachko added a comment - This sounds like a good optimization for viewfs and a pretty clean patch. Will commit it soon, unless other people want to review it as well. +1
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for working on this Erik! I took a quick look at the patch, seems reasonable, +0 from me.

          I also retriggered the precommit build so we can get a run before committing.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for working on this Erik! I took a quick look at the patch, seems reasonable, +0 from me. I also retriggered the precommit build so we can get a run before committing.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 from me also, pending pre-commit run on the latest revision. Thank you, Erik Krogen.

          Show
          cnauroth Chris Nauroth added a comment - +1 from me also, pending pre-commit run on the latest revision. Thank you, Erik Krogen .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 34s trunk passed
          +1 compile 11m 42s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 14s trunk passed
          +1 mvneclipse 0m 21s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 54s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 12m 6s the patch passed
          +1 javac 12m 6s the patch passed
          -0 checkstyle 0m 42s hadoop-common-project/hadoop-common: The patch generated 1 new + 164 unchanged - 7 fixed = 165 total (was 171)
          +1 mvnsite 1m 6s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 42s the patch passed
          +1 javadoc 0m 55s the patch passed
          -1 unit 8m 25s hadoop-common in the patch failed.
          +1 asflicense 0m 43s The patch does not generate ASF License warnings.
          58m 37s



          Reason Tests
          Failed junit tests hadoop.fs.viewfs.TestViewFsTrash
            hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14169
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857395/HADOOP-14169.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ee39e279f417 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 615ac09
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11831/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11831/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11831/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11831/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 34s trunk passed +1 compile 11m 42s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 14s trunk passed +1 mvneclipse 0m 21s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 12m 6s the patch passed +1 javac 12m 6s the patch passed -0 checkstyle 0m 42s hadoop-common-project/hadoop-common: The patch generated 1 new + 164 unchanged - 7 fixed = 165 total (was 171) +1 mvnsite 1m 6s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 42s the patch passed +1 javadoc 0m 55s the patch passed -1 unit 8m 25s hadoop-common in the patch failed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 58m 37s Reason Tests Failed junit tests hadoop.fs.viewfs.TestViewFsTrash   hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14169 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12857395/HADOOP-14169.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ee39e279f417 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 615ac09 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11831/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11831/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11831/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11831/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          Looks like TestViewFsTrash has been failing intermittently on Jenkins builds recently; passes fine locally.

          Show
          xkrogen Erik Krogen added a comment - Looks like TestViewFsTrash has been failing intermittently on Jenkins builds recently; passes fine locally.
          Hide
          shv Konstantin Shvachko added a comment -

          I just committed this. Thank you Erik Krogen.

          Show
          shv Konstantin Shvachko added a comment - I just committed this. Thank you Erik Krogen .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11424 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11424/)
          HADOOP-14169. Implement listStatusIterator, listLocatedStatus for (shv: rev e1a99802fc5c2f2ad3ceb000378db1c0133069b3)

          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFs.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11424 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11424/ ) HADOOP-14169 . Implement listStatusIterator, listLocatedStatus for (shv: rev e1a99802fc5c2f2ad3ceb000378db1c0133069b3) (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFs.java
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

            People

            • Assignee:
              xkrogen Erik Krogen
              Reporter:
              xkrogen Erik Krogen
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development