Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: datanode, fs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change

      Description

      Modifications to FsVolumeSpi and FsVolumeImpl to remove references to java.io.File.

      1. HDFS-10637.011.patch
        148 kB
        Virajith Jalaparti
      2. HDFS-10637.010.patch
        148 kB
        Virajith Jalaparti
      3. HDFS-10637.009.patch
        149 kB
        Virajith Jalaparti
      4. HDFS-10637.008.patch
        150 kB
        Virajith Jalaparti
      5. HDFS-10637.007.patch
        150 kB
        Virajith Jalaparti
      6. HDFS-10637.006.patch
        146 kB
        Virajith Jalaparti
      7. HDFS-10637.005.patch
        146 kB
        Virajith Jalaparti
      8. HDFS-10637.004.patch
        147 kB
        Virajith Jalaparti
      9. HDFS-10637.003.patch
        150 kB
        Virajith Jalaparti
      10. HDFS-10637.002.patch
        146 kB
        Virajith Jalaparti
      11. HDFS-10637.001.patch
        150 kB
        Virajith Jalaparti

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Similar question here as to on HDFS-10636, is this really an incompatible change? Seems to only modify internal APIs.

          Show
          andrew.wang Andrew Wang added a comment - Similar question here as to on HDFS-10636 , is this really an incompatible change? Seems to only modify internal APIs.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the response Virajith Jalaparti. No use case from me - just wanted to know. The divergence between trunk and branch-2 will complicate backporting later DataNode fixes to 2.x but it was inevitable.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the response Virajith Jalaparti . No use case from me - just wanted to know. The divergence between trunk and branch-2 will complicate backporting later DataNode fixes to 2.x but it was inevitable.
          Hide
          virajith Virajith Jalaparti added a comment -

          Hi Arpit Agarwal, We currently do not have a plan to port these changes back to branch-2. However, if this would be useful, we can do it. Will you have specific use cases where you might want to use PROVIDED storage in branch-2 as opposed to 3?

          Show
          virajith Virajith Jalaparti added a comment - Hi Arpit Agarwal , We currently do not have a plan to port these changes back to branch-2. However, if this would be useful, we can do it. Will you have specific use cases where you might want to use PROVIDED storage in branch-2 as opposed to 3?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Virajith Jalaparti, Lei (Eddy) Xu,

          Do you see these changes going to branch-2 eventually or will they be 3.x-only?

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Virajith Jalaparti , Lei (Eddy) Xu , Do you see these changes going to branch-2 eventually or will they be 3.x-only?
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10583 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10583/)
          HDFS-10637. Modifications to remove the assumption that FsVolumes are (lei: rev 96b12662ea76e3ded4ef13944fc8df206cfb4613)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/TestDiskBalancerWithMockMover.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockScanner.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/VolumeFailureInfo.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskBalancer.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockScanner.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
          • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImplBuilder.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplica.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10583 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10583/ ) HDFS-10637 . Modifications to remove the assumption that FsVolumes are (lei: rev 96b12662ea76e3ded4ef13944fc8df206cfb4613) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/diskbalancer/TestDiskBalancerWithMockMover.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockScanner.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/VolumeFailureInfo.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DiskBalancer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockScanner.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImplBuilder.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplica.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          +1. The checkstyle warnings were existed ones, caused by moving code around.

          Committed to trunk.

          Thanks for the great work, Virajith Jalaparti.

          Show
          eddyxu Lei (Eddy) Xu added a comment - +1. The checkstyle warnings were existed ones, caused by moving code around. Committed to trunk. Thanks for the great work, Virajith Jalaparti .
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 14 new or modified test files.
          +1 mvninstall 7m 17s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 38s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          -0 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 984 unchanged - 11 fixed = 993 total (was 995)
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 51s the patch passed
          +1 javadoc 0m 52s the patch passed
          +1 unit 61m 27s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          81m 32s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10637
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831807/HDFS-10637.011.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 04098573a30c 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 00160f7
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17026/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17026/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17026/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 14 new or modified test files. +1 mvninstall 7m 17s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 984 unchanged - 11 fixed = 993 total (was 995) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 51s the patch passed +1 javadoc 0m 52s the patch passed +1 unit 61m 27s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 81m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10637 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831807/HDFS-10637.011.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 04098573a30c 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 00160f7 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17026/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17026/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17026/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          virajith Virajith Jalaparti added a comment -

          New patch to fix the failing test case in TestDirectoryScanner.

          Show
          virajith Virajith Jalaparti added a comment - New patch to fix the failing test case in TestDirectoryScanner .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 14 new or modified test files.
          +1 mvninstall 7m 11s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 984 unchanged - 11 fixed = 993 total (was 995)
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 54s the patch passed
          -1 unit 60m 6s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          80m 13s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.TestRenameWhileOpen



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10637
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831203/HDFS-10637.010.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 89dc4e911088 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 89bd6d2
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16963/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16963/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16963/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16963/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 14 new or modified test files. +1 mvninstall 7m 11s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 984 unchanged - 11 fixed = 993 total (was 995) +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 60m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 80m 13s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestRenameWhileOpen Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10637 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831203/HDFS-10637.010.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 89dc4e911088 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 89bd6d2 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16963/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16963/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16963/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16963/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          virajith Virajith Jalaparti added a comment -

          Attaching new patch based on Lei (Eddy) Xu's comments.

          Show
          virajith Virajith Jalaparti added a comment - Attaching new patch based on Lei (Eddy) Xu 's comments.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Yes. That's right.

          Thanks a lot!

          Show
          eddyxu Lei (Eddy) Xu added a comment - Yes. That's right. Thanks a lot!
          Hide
          virajith Virajith Jalaparti added a comment -

          Hi Lei (Eddy) Xu,

          what about setting FsVolumeImpl#storageStorage as final ...

          I assume you meant FsVolumeImpl#storageLocation here? Yes, I think this makes sense. So, the changes will be the following:
          1) Have only one constructor for FsVolumeImpl, which takes in as input the StorageDirectory. This would involve moving the code in the private FsVolumeImpl(...String currentDir...StorageLocation storageLocation) to the FsVolumeImpl(...,StorageDirectory sd, ...) constructor.

          2) Declare both FsVolumeImpl#storageLocation, and FsVolumeImpl#currentDir as final.

          The above changes will ensure that both are initialized consistently, and they are not modified again.

          Is this what you had in mind?

          I will post the updated patch soon.

          Show
          virajith Virajith Jalaparti added a comment - Hi Lei (Eddy) Xu , what about setting FsVolumeImpl#storageStorage as final ... I assume you meant FsVolumeImpl#storageLocation here? Yes, I think this makes sense. So, the changes will be the following: 1) Have only one constructor for FsVolumeImpl , which takes in as input the StorageDirectory . This would involve moving the code in the private FsVolumeImpl(...String currentDir...StorageLocation storageLocation) to the FsVolumeImpl(...,StorageDirectory sd, ...) constructor. 2) Declare both FsVolumeImpl#storageLocation , and FsVolumeImpl#currentDir as final. The above changes will ensure that both are initialized consistently, and they are not modified again. Is this what you had in mind? I will post the updated patch soon.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Virajith Jalaparti Thanks a lot for update the patch. It looks really good.

          currentDir is now just a private variable that holds this value.

          My only concerns is that, because currentDir == storageLocation + STORAGE_DIR_CURRENT, putting both of them into FsVolumeImpl, introduces the chance of inconsistent states (i.e., currentDir != storageLocation + /current). Alternatively, what about setting FsVolumeImpl#storageStorage as final, and do not introduce private FsVolumeImpl(...String currentDir...StorageLocation storageLocation), which will likely become public later by other people. So that you can consider currentDir is a cached value of storageLocation or vice versa.

          What do you think?

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Virajith Jalaparti Thanks a lot for update the patch. It looks really good. currentDir is now just a private variable that holds this value. My only concerns is that, because currentDir == storageLocation + STORAGE_DIR_CURRENT , putting both of them into FsVolumeImpl , introduces the chance of inconsistent states (i.e., currentDir != storageLocation + /current ). Alternatively, what about setting FsVolumeImpl#storageStorage as final , and do not introduce private FsVolumeImpl(...String currentDir...StorageLocation storageLocation) , which will likely become public later by other people. So that you can consider currentDir is a cached value of storageLocation or vice versa. What do you think?
          Hide
          virajith Virajith Jalaparti added a comment -

          Thanks Lei (Eddy) Xu for the clarification. I am posting a new patch to address your comments. I wanted to point out two in particular:

          • Re FsDatasetSpi#getFileInFinalizedDir() - I removed this. In the test cases where this was used (e.g. TestDirectoryScanner), I replaced the call with FsVolumeImpl#getFinalizedDir. FsVolumeImpl#getFinalizedDir is currently used only in the test cases, and is marked as such. Another way to address this (given the way FsVolumeImpl#getFinalizedDir is used in the test cases), could have been to add calls such as createBlock and createBlockMetadata to FsVolumeImpl, which create files for the block data and block metadata (respectively). The former re-uses existing functions, and seems cleaner.
          • Re currentDir in FsVolumeImpl - I have not removed this yet. It is a private variable of FsVolumeImpl. I think having it makes the code simpler – if not, any use of currentDir needs to be replaced by something like new File(StorageLocation.getFile(), STORAGE_DIR_CURRENT). currentDir is now just a private variable that holds this value. What do you think?
          Show
          virajith Virajith Jalaparti added a comment - Thanks Lei (Eddy) Xu for the clarification. I am posting a new patch to address your comments. I wanted to point out two in particular: Re FsDatasetSpi#getFileInFinalizedDir() - I removed this. In the test cases where this was used (e.g. TestDirectoryScanner ), I replaced the call with FsVolumeImpl#getFinalizedDir . FsVolumeImpl#getFinalizedDir is currently used only in the test cases, and is marked as such. Another way to address this (given the way FsVolumeImpl#getFinalizedDir is used in the test cases), could have been to add calls such as createBlock and createBlockMetadata to FsVolumeImpl , which create files for the block data and block metadata (respectively). The former re-uses existing functions, and seems cleaner. Re currentDir in FsVolumeImpl - I have not removed this yet. It is a private variable of FsVolumeImpl . I think having it makes the code simpler – if not, any use of currentDir needs to be replaced by something like new File(StorageLocation.getFile(), STORAGE_DIR_CURRENT) . currentDir is now just a private variable that holds this value. What do you think?
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Can you confirm which one are you referring to?

          Sure. I mean you can remove FsVolumeImpl#currentDir as a class member, as it can be derived from FsVolumeImpl#storageLocation.

          FsDatasetSpi#getFileInFinalizedDir().

          It still assumes FsDatasetSpi is backed by java.io.File. If this is only used by tests, can you use FsDatasetTestUtils#getBlockFile and #getMetaFile, as they are in the test package. So that FsDatasetSpi is clean from java.io.File?

          Thanks.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Can you confirm which one are you referring to? Sure. I mean you can remove FsVolumeImpl#currentDir as a class member, as it can be derived from FsVolumeImpl#storageLocation . FsDatasetSpi#getFileInFinalizedDir(). It still assumes FsDatasetSpi is backed by java.io.File . If this is only used by tests, can you use FsDatasetTestUtils#getBlockFile and #getMetaFile , as they are in the test package. So that FsDatasetSpi is clean from java.io.File ? Thanks.
          Hide
          virajith Virajith Jalaparti added a comment -

          Hi Lei (Eddy) Xu,

          A couple of questions:

          In FsVolumeImpl can you remove this.currentDir?

          Can you confirm which one are you referring to? In the constructor? If so, is the intent to change the name of the argument passed into the constructor?

          FsDatasetSpi#getFileInFinalizedDir(), it still returns a File. Is there a way to eliminate it?

          It is used only in TestDirectoryScanner. One way to addressed this is to use FsVolumeImpl.getFinalizedDir as before. This function is marked @VisibleForTesting, and is used in a couple of other tests. Another way would be essentially rewrite parts of the TestDirectoryScanner which use this function. In this case, some of the tests (e.g., TestScanInfo) do not make sense as they rely on a File being created for a block, and then check if the file exists or not. What do you think?

          Show
          virajith Virajith Jalaparti added a comment - Hi Lei (Eddy) Xu , A couple of questions: In FsVolumeImpl can you remove this.currentDir ? Can you confirm which one are you referring to? In the constructor? If so, is the intent to change the name of the argument passed into the constructor? FsDatasetSpi#getFileInFinalizedDir() , it still returns a File. Is there a way to eliminate it? It is used only in TestDirectoryScanner . One way to addressed this is to use FsVolumeImpl.getFinalizedDir as before. This function is marked @VisibleForTesting , and is used in a couple of other tests. Another way would be essentially rewrite parts of the TestDirectoryScanner which use this function. In this case, some of the tests (e.g., TestScanInfo ) do not make sense as they rely on a File being created for a block, and then check if the file exists or not. What do you think?
          Hide
          virajith Virajith Jalaparti added a comment -

          Thanks for the comments Lei (Eddy) Xu. I will address them in the next patch.

          Show
          virajith Virajith Jalaparti added a comment - Thanks for the comments Lei (Eddy) Xu . I will address them in the next patch.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          HI, Virajith Jalaparti

          Thanks for the patch.

          • Please add ASF license to FsVolumeImplBuilder
          • TestNameNodePrunesMissingStorages.java:
                StorageLocation volumeDirectoryToRemove = null;
          

          Change to volumeLocationToRemove?

          FileUtil.fullyDelete(
          237	          new File(volumeDirectoryToRemove.getFile().toString()));
          

          getFile() already returns a File. you dont need to create a new one.

          • In FsVolumeImpl. can you remove this.currentDir?
          • FsDatasetSpi#getFileInFinalizedDir(), it still returns a File. Is there a way to eliminate it?
          • FsVolumeSpi#containsPath is only used by test? I feel that it is still assuming a file based FsVolume.

          It is a large patch. Please allow more comments to come later. Thanks.

          Show
          eddyxu Lei (Eddy) Xu added a comment - HI, Virajith Jalaparti Thanks for the patch. Please add ASF license to FsVolumeImplBuilder TestNameNodePrunesMissingStorages.java : StorageLocation volumeDirectoryToRemove = null ; Change to volumeLocationToRemove ? FileUtil.fullyDelete( 237 new File(volumeDirectoryToRemove.getFile().toString())); getFile() already returns a File . you dont need to create a new one. In FsVolumeImpl . can you remove this.currentDir ? FsDatasetSpi#getFileInFinalizedDir() , it still returns a File. Is there a way to eliminate it? FsVolumeSpi#containsPath is only used by test? I feel that it is still assuming a file based FsVolume . It is a large patch. Please allow more comments to come later. Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 14 new or modified test files.
          +1 mvninstall 7m 49s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 38s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 47s trunk passed
          +1 javadoc 1m 3s trunk passed
          +1 mvninstall 0m 54s the patch passed
          +1 compile 0m 49s the patch passed
          +1 javac 0m 49s the patch passed
          -0 checkstyle 0m 37s hadoop-hdfs-project/hadoop-hdfs: The patch generated 11 new + 998 unchanged - 11 fixed = 1009 total (was 1009)
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 1m 0s the patch passed
          -1 unit 64m 3s hadoop-hdfs in the patch failed.
          -1 asflicense 0m 19s The patch generated 1 ASF License warnings.
          85m 48s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10637
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828541/HDFS-10637.008.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2a0717c63288 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2a8f55a
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16746/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16746/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16746/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16746/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16746/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 14 new or modified test files. +1 mvninstall 7m 49s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 47s trunk passed +1 javadoc 1m 3s trunk passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 49s the patch passed +1 javac 0m 49s the patch passed -0 checkstyle 0m 37s hadoop-hdfs-project/hadoop-hdfs: The patch generated 11 new + 998 unchanged - 11 fixed = 1009 total (was 1009) +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 1m 0s the patch passed -1 unit 64m 3s hadoop-hdfs in the patch failed. -1 asflicense 0m 19s The patch generated 1 ASF License warnings. 85m 48s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10637 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828541/HDFS-10637.008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2a0717c63288 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2a8f55a Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16746/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16746/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16746/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16746/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16746/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 14 new or modified test files.
          +1 mvninstall 8m 36s trunk passed
          +1 compile 1m 0s trunk passed
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 1m 9s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 0s trunk passed
          +1 javadoc 1m 3s trunk passed
          +1 mvninstall 1m 4s the patch passed
          +1 compile 0m 56s the patch passed
          +1 javac 0m 56s the patch passed
          -0 checkstyle 0m 41s hadoop-hdfs-project/hadoop-hdfs: The patch generated 10 new + 999 unchanged - 11 fixed = 1009 total (was 1010)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 51s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 55s the patch passed
          -1 unit 77m 47s hadoop-hdfs in the patch failed.
          -1 asflicense 0m 18s The patch generated 1 ASF License warnings.
          101m 10s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            FsVolumeImpl is incompatible with expected argument type String in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.addVolume(FsVolumeImpl) At FsDatasetAsyncDiskService.java:argument type String in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.addVolume(FsVolumeImpl) At FsDatasetAsyncDiskService.java:[line 129]
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10637
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828509/HDFS-10637.007.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0ece8f984bfe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2a8f55a
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16741/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16741/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16741/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16741/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16741/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 14 new or modified test files. +1 mvninstall 8m 36s trunk passed +1 compile 1m 0s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 1m 9s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 3s trunk passed +1 mvninstall 1m 4s the patch passed +1 compile 0m 56s the patch passed +1 javac 0m 56s the patch passed -0 checkstyle 0m 41s hadoop-hdfs-project/hadoop-hdfs: The patch generated 10 new + 999 unchanged - 11 fixed = 1009 total (was 1010) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 51s hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 55s the patch passed -1 unit 77m 47s hadoop-hdfs in the patch failed. -1 asflicense 0m 18s The patch generated 1 ASF License warnings. 101m 10s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   FsVolumeImpl is incompatible with expected argument type String in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.addVolume(FsVolumeImpl) At FsDatasetAsyncDiskService.java:argument type String in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.addVolume(FsVolumeImpl) At FsDatasetAsyncDiskService.java: [line 129] Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10637 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828509/HDFS-10637.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0ece8f984bfe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2a8f55a Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16741/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16741/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/16741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16741/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16741/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16741/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          virajith Virajith Jalaparti added a comment -

          Updated patch to work with the most recent patch of HDFS-10636

          Show
          virajith Virajith Jalaparti added a comment - Updated patch to work with the most recent patch of HDFS-10636
          Hide
          virajith Virajith Jalaparti added a comment -

          Posting an updated patch (to work with the most recent changes in HDFS-10636).

          Show
          virajith Virajith Jalaparti added a comment - Posting an updated patch (to work with the most recent changes in HDFS-10636 ).
          Hide
          virajith Virajith Jalaparti added a comment -

          Posting a modified patch to work with the latest patch on HDFS-10636.

          Show
          virajith Virajith Jalaparti added a comment - Posting a modified patch to work with the latest patch on HDFS-10636 .
          Hide
          virajith Virajith Jalaparti added a comment -

          New patch to account for changes in the patch of HDFS-10636.

          Show
          virajith Virajith Jalaparti added a comment - New patch to account for changes in the patch of HDFS-10636 .
          Hide
          virajith Virajith Jalaparti added a comment -

          Posting a new patch to work with the new patch of HDFS-10636.

          Show
          virajith Virajith Jalaparti added a comment - Posting a new patch to work with the new patch of HDFS-10636 .
          Hide
          virajith Virajith Jalaparti added a comment - - edited

          The patch contains the following changes (text is incorporated from comments in HDFS-9809):

          1. Instead of associating an FsVolume with a base path (which is a java.io.File), we associate it with a StorageLocation. This allows removing the dependence on java.io.File and replacing it with a more general one, which can point to a java.io.File or an abstract URI representing an external storage. Using StorageLocation instead of defining a new type for location allows us to reuse its functionality and plug into the rest of the code easily. Following this intuition, we replaced FsVolumeSpi#getBasePath with FsVolumeSpi#getStorageLocation. As a result, comparisons and references to FsVolumes which were done using the java.io.File returned by FsVolumeSpi#getBasePath are now replaced by comparisons and references to the StorageLocation returned by FsVolumeSpi#getStorageLocation.
          2. Similarly, the patch also replaces calls to get*Dir on FsVolumeImpl. In particular, the DirectoryScanner.ReportCompiler calls on FsVolumeSpi#getFinalizedDir and compiles the report assuming that this returns a java.io.File. However, in HDFS-9806, data may not be stored in files. Further, the DirectoryScanner.ReportCompiler#compileReport function assumes the way blocks are stored in FsVolumes which can be different for different FsVolumeSpi implementations. To address these assumptions and to hide the details on how volumes implement block storage, ReportCompiler#compileReport is moved to FsVolumeSpi.
          3. A FsVolumeImplBuilder is added and calls to the constructor of FsVolumeImpl are replaced with those to the builder. The idea behind this is to construct the appropriate volume based on the StorageLocation (specified using dfs.datanode.data.dir). For example, as part of HDFS-9806, if a StorageLocation is of a PROVIDED type, we would construct a ProvidedVolumeImpl. Otherwise, a FsVolumeImpl would be built.
          Show
          virajith Virajith Jalaparti added a comment - - edited The patch contains the following changes (text is incorporated from comments in HDFS-9809 ): Instead of associating an FsVolume with a base path (which is a java.io.File ), we associate it with a StorageLocation . This allows removing the dependence on java.io.File and replacing it with a more general one, which can point to a java.io.File or an abstract URI representing an external storage. Using StorageLocation instead of defining a new type for location allows us to reuse its functionality and plug into the rest of the code easily. Following this intuition, we replaced FsVolumeSpi#getBasePath with FsVolumeSpi#getStorageLocation . As a result, comparisons and references to FsVolumes which were done using the java.io.File returned by FsVolumeSpi#getBasePath are now replaced by comparisons and references to the StorageLocation returned by FsVolumeSpi#getStorageLocation . Similarly, the patch also replaces calls to get*Dir on FsVolumeImpl. In particular, the DirectoryScanner.ReportCompiler calls on FsVolumeSpi#getFinalizedDir and compiles the report assuming that this returns a java.io.File . However, in HDFS-9806 , data may not be stored in files. Further, the DirectoryScanner.ReportCompiler#compileReport function assumes the way blocks are stored in FsVolumes which can be different for different FsVolumeSpi implementations. To address these assumptions and to hide the details on how volumes implement block storage, ReportCompiler#compileReport is moved to FsVolumeSpi . A FsVolumeImplBuilder is added and calls to the constructor of FsVolumeImpl are replaced with those to the builder. The idea behind this is to construct the appropriate volume based on the StorageLocation (specified using dfs.datanode.data.dir ). For example, as part of HDFS-9806 , if a StorageLocation is of a PROVIDED type, we would construct a ProvidedVolumeImpl . Otherwise, a FsVolumeImpl would be built.

            People

            • Assignee:
              virajith Virajith Jalaparti
              Reporter:
              virajith Virajith Jalaparti
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development