Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Some code cleanup for DatanodeDescriptor and HeartbeatManager. The changes include:

      1. Change DataDescriptor#isAlive and DatanodeDescriptor#needKeyUpdate from public to private
      2. Use EnumMap for HeartbeatManager#storageTypeStatesMap
      3. Move the isInStartupSafeMode out of the namesystem lock in heartbeatCheck
      1. HDFS-9223.003.patch
        44 kB
        Jing Zhao
      2. HDFS-9223.002.patch
        43 kB
        Jing Zhao
      3. HDFS-9223.001.patch
        43 kB
        Jing Zhao
      4. HDFS-9223.000.patch
        30 kB
        Jing Zhao

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #498 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/498/)
        HDFS-9223. Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #498 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/498/ ) HDFS-9223 . Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2435 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2435/)
        HDFS-9223. Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2435 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2435/ ) HDFS-9223 . Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2481 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2481/)
        HDFS-9223. Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2481 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2481/ ) HDFS-9223 . Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #532 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/532/)
        HDFS-9223. Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #532 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/532/ ) HDFS-9223 . Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #545 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/545/)
        HDFS-9223. Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #545 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/545/ ) HDFS-9223 . Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1268 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1268/)
        HDFS-9223. Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1268 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1268/ ) HDFS-9223 . Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8638 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8638/)
        HDFS-9223. Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8638 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8638/ ) HDFS-9223 . Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9: rev be7a0add8b6561d3c566237cc0370b06e7f32bb4) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockUnderConstructionFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
        Hide
        jingzhao Jing Zhao added a comment -

        Thanks again for the review, Nicholas! I've updated the patch based on your comments and committed it to trunk and branch-2.8.

        Show
        jingzhao Jing Zhao added a comment - Thanks again for the review, Nicholas! I've updated the patch based on your comments and committed it to trunk and branch-2.8.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 19m 30s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 8m 4s There were no new javac warning messages.
        +1 javadoc 10m 29s There were no new javadoc warning messages.
        -1 release audit 0m 19s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 24s The applied patch generated 4 new checkstyle issues (total was 328, now 318).
        -1 whitespace 0m 3s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 29s mvn install still works.
        +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
        +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 14s Pre-build of native portion
        -1 hdfs tests 56m 24s Tests failed in hadoop-hdfs.
            104m 9s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestRecoverStripedFile
          hadoop.fs.TestGlobPaths



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12766595/HDFS-9223.003.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / ba3c197
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12988/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12988/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12988/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12988/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12988/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12988/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12988/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 19m 30s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 8m 4s There were no new javac warning messages. +1 javadoc 10m 29s There were no new javadoc warning messages. -1 release audit 0m 19s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 24s The applied patch generated 4 new checkstyle issues (total was 328, now 318). -1 whitespace 0m 3s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 56m 24s Tests failed in hadoop-hdfs.     104m 9s   Reason Tests Failed unit tests hadoop.hdfs.TestRecoverStripedFile   hadoop.fs.TestGlobPaths Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766595/HDFS-9223.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ba3c197 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12988/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12988/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12988/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12988/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12988/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12988/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12988/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 16m 32s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 8m 1s There were no new javac warning messages.
        +1 javadoc 10m 42s There were no new javadoc warning messages.
        -1 release audit 0m 23s The applied patch generated 1 release audit warnings.
        +1 checkstyle 0m 47s There were no new checkstyle issues.
        -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 39s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        -1 findbugs 0m 28s Post-patch findbugs hadoop-hdfs-project/hadoop-hdfs compilation is broken.
        +1 findbugs 0m 28s The patch does not introduce any new Findbugs (version ) warnings.
        -1 native 0m 26s Failed to build the native portion of hadoop-common prior to running the unit tests in hadoop-hdfs-project/hadoop-hdfs
            39m 37s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12766595/HDFS-9223.003.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 0d77e85
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12982/artifact/patchprocess/patchReleaseAuditProblems.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12982/artifact/patchprocess/whitespace.txt
        Java 1.7.0_55
        uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12982/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 32s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 8m 1s There were no new javac warning messages. +1 javadoc 10m 42s There were no new javadoc warning messages. -1 release audit 0m 23s The applied patch generated 1 release audit warnings. +1 checkstyle 0m 47s There were no new checkstyle issues. -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 0m 28s Post-patch findbugs hadoop-hdfs-project/hadoop-hdfs compilation is broken. +1 findbugs 0m 28s The patch does not introduce any new Findbugs (version ) warnings. -1 native 0m 26s Failed to build the native portion of hadoop-common prior to running the unit tests in hadoop-hdfs-project/hadoop-hdfs     39m 37s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766595/HDFS-9223.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0d77e85 Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12982/artifact/patchprocess/patchReleaseAuditProblems.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12982/artifact/patchprocess/whitespace.txt Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12982/console This message was automatically generated.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        Still need to change getCapacityUsedPercent.

        +1 after it is addressed.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - Still need to change getCapacityUsedPercent. +1 after it is addressed.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 18m 21s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 10m 0s There were no new javac warning messages.
        +1 javadoc 12m 41s There were no new javadoc warning messages.
        -1 release audit 0m 20s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 28s The applied patch generated 4 new checkstyle issues (total was 329, now 319).
        -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 45s mvn install still works.
        +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse.
        +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 4m 17s Pre-build of native portion
        -1 hdfs tests 57m 38s Tests failed in hadoop-hdfs.
            110m 20s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.TestReplaceDatanodeOnFailure
          hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
        Timed out tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
          org.apache.hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
          org.apache.hadoop.hdfs.server.datanode.TestBPOfferService



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12766371/HDFS-9223.002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / aa299ec
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12961/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12961/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12961/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12961/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12961/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12961/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12961/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 21s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 10m 0s There were no new javac warning messages. +1 javadoc 12m 41s There were no new javadoc warning messages. -1 release audit 0m 20s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 28s The applied patch generated 4 new checkstyle issues (total was 329, now 319). -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 45s mvn install still works. +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse. +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 4m 17s Pre-build of native portion -1 hdfs tests 57m 38s Tests failed in hadoop-hdfs.     110m 20s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints Timed out tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement   org.apache.hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   org.apache.hadoop.hdfs.server.datanode.TestBPOfferService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766371/HDFS-9223.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / aa299ec Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12961/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12961/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12961/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12961/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12961/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12961/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12961/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 18m 41s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 8m 20s There were no new javac warning messages.
        +1 javadoc 10m 33s There were no new javadoc warning messages.
        -1 release audit 0m 19s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 25s The applied patch generated 4 new checkstyle issues (total was 329, now 319).
        -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 32s mvn install still works.
        +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
        +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 16s Pre-build of native portion
        -1 hdfs tests 64m 14s Tests failed in hadoop-hdfs.
            111m 32s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.blockmanagement.TestNodeCount
          hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.TestRollingUpgrade



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12766371/HDFS-9223.002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / da16c9b
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12956/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12956/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12956/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12956/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12956/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12956/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12956/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 41s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 8m 20s There were no new javac warning messages. +1 javadoc 10m 33s There were no new javadoc warning messages. -1 release audit 0m 19s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 25s The applied patch generated 4 new checkstyle issues (total was 329, now 319). -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 16s Pre-build of native portion -1 hdfs tests 64m 14s Tests failed in hadoop-hdfs.     111m 32s   Reason Tests Failed unit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestRollingUpgrade Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766371/HDFS-9223.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / da16c9b Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12956/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12956/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12956/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12956/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12956/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12956/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12956/console This message was automatically generated.
        Hide
        jingzhao Jing Zhao added a comment -

        Update the patch to address Nicholas's comments.

        Show
        jingzhao Jing Zhao added a comment - Update the patch to address Nicholas's comments.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        Since getCapacityUsedPercent, getCapacityRemainingPercent, getPercentBlockPoolUsed, getCapacityUsedNonDFS call multiple stats methods, we should also add them to the new class and synchronize them.

        > ... I will check this code to confirm and maybe we can do this change separately.

        Sure, please check the code and do it separately.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - Since getCapacityUsedPercent, getCapacityRemainingPercent, getPercentBlockPoolUsed, getCapacityUsedNonDFS call multiple stats methods, we should also add them to the new class and synchronize them. > ... I will check this code to confirm and maybe we can do this change separately. Sure, please check the code and do it separately.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 24m 28s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 11m 9s There were no new javac warning messages.
        +1 javadoc 11m 45s There were no new javadoc warning messages.
        -1 release audit 0m 21s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 46s The applied patch generated 4 new checkstyle issues (total was 330, now 320).
        -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 41s mvn install still works.
        +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
        +1 findbugs 2m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 35s Pre-build of native portion
        -1 hdfs tests 197m 42s Tests failed in hadoop-hdfs.
            256m 1s  



        Reason Tests
        Failed unit tests hadoop.hdfs.util.TestByteArrayManager



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12766169/HDFS-9223.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 9849c8b
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12936/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12936/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12936/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12936/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12936/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12936/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 24m 28s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 11m 9s There were no new javac warning messages. +1 javadoc 11m 45s There were no new javadoc warning messages. -1 release audit 0m 21s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 46s The applied patch generated 4 new checkstyle issues (total was 330, now 320). -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 2m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 35s Pre-build of native portion -1 hdfs tests 197m 42s Tests failed in hadoop-hdfs.     256m 1s   Reason Tests Failed unit tests hadoop.hdfs.util.TestByteArrayManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766169/HDFS-9223.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9849c8b Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12936/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12936/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12936/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12936/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12936/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12936/console This message was automatically generated.
        Hide
        jingzhao Jing Zhao added a comment -

        For heartbeatCheck, currently the dead node object is retrieved from the heartbeatManager's own datanode list, I'm not very sure if this list is always consistent with datanodeManager's datanodeMap. I will check this code to confirm and maybe we can do this change separately.

        Show
        jingzhao Jing Zhao added a comment - For heartbeatCheck , currently the dead node object is retrieved from the heartbeatManager's own datanode list, I'm not very sure if this list is always consistent with datanodeManager's datanodeMap. I will check this code to confirm and maybe we can do this change separately.
        Hide
        jingzhao Jing Zhao added a comment -

        Thanks for the review, Nicholas! Update patch to address your comments. Also I moved the stats related code into a separate class and made it thread safe.

        Show
        jingzhao Jing Zhao added a comment - Thanks for the review, Nicholas! Update patch to address your comments. Also I moved the stats related code into a separate class and made it thread safe.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        > ... do you think this change is OK?

        Yes, the change looks good.

        Some minor comments:

        • Please as well fix the "atleast" typo in the javadoc of DatanodeDescriptor.checkBlockReportReceived().
        • It looks like the following format change was done by accident.
          -    BlockManager.LOG.debug("Adding block recovery task " + task + "to "
          -        + getName() + ", current queue size is " + erasurecodeBlocks.size());
          +    BlockManager.LOG.debug("Adding block recovery task " + task + "to " + getName() + ", current queue size is " + erasurecodeBlocks.size());
          

        Other suggestions:

        • The stats related calls should synchronize on HeartbeatManager.stats but not the HeartbeatManager object.
        • In heartbeatCheck(), the local variable dead can be changed from DatanodeID to DatanodeDescriptor. Then, could we as well change DatanodeManager.removeDeadDatanode(DatanodeID) to DatanodeManager.removeDeadDatanode(DatanodeDescriptor) so that it does not need to lookup the DatanodeDescriptor?
        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - > ... do you think this change is OK? Yes, the change looks good. Some minor comments: Please as well fix the "atleast" typo in the javadoc of DatanodeDescriptor.checkBlockReportReceived(). It looks like the following format change was done by accident. - BlockManager.LOG.debug( "Adding block recovery task " + task + "to " - + getName() + ", current queue size is " + erasurecodeBlocks.size()); + BlockManager.LOG.debug( "Adding block recovery task " + task + "to " + getName() + ", current queue size is " + erasurecodeBlocks.size()); Other suggestions: The stats related calls should synchronize on HeartbeatManager.stats but not the HeartbeatManager object. In heartbeatCheck(), the local variable dead can be changed from DatanodeID to DatanodeDescriptor. Then, could we as well change DatanodeManager.removeDeadDatanode(DatanodeID) to DatanodeManager.removeDeadDatanode(DatanodeDescriptor) so that it does not need to lookup the DatanodeDescriptor?
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 20m 26s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 4 new or modified test files.
        +1 javac 8m 56s There were no new javac warning messages.
        +1 javadoc 11m 54s There were no new javadoc warning messages.
        -1 release audit 0m 22s The applied patch generated 1 release audit warnings.
        -1 checkstyle 1m 37s The applied patch generated 5 new checkstyle issues (total was 331, now 323).
        -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
        +1 findbugs 2m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 40s Pre-build of native portion
        -1 hdfs tests 246m 49s Tests failed in hadoop-hdfs.
            298m 54s  



        Reason Tests
        Failed unit tests hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.hdfs.server.namenode.TestFSNamesystem
        Timed out tests org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12765921/HDFS-9223.000.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / def374e
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12901/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12901/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12901/artifact/patchprocess/whitespace.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12901/artifact/patchprocess/testrun_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12901/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12901/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 20m 26s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 4 new or modified test files. +1 javac 8m 56s There were no new javac warning messages. +1 javadoc 11m 54s There were no new javadoc warning messages. -1 release audit 0m 22s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 37s The applied patch generated 5 new checkstyle issues (total was 331, now 323). -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 2m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 40s Pre-build of native portion -1 hdfs tests 246m 49s Tests failed in hadoop-hdfs.     298m 54s   Reason Tests Failed unit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.namenode.TestFSNamesystem Timed out tests org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765921/HDFS-9223.000.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / def374e Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12901/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12901/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12901/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12901/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12901/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12901/console This message was automatically generated.
        Hide
        jingzhao Jing Zhao added a comment -

        The patch also makes the following changes in heartbeatCheck.

        -          synchronized(this) {
        -            dm.removeDeadDatanode(dead);
        -          }
        +          dm.removeDeadDatanode(dead);
        
        -          synchronized(this) {
        -            blockManager.removeBlocksAssociatedTo(failedStorage);
        -          }
        +          blockManager.removeBlocksAssociatedTo(failedStorage);
        

        Looks like the changes to heartbeat#datanodes are already protected by the heartbeat manager's monitor in corresponding methods. The heartbeat monitor was added originally by HDFS-2108. Tsz Wo Nicholas Sze, do you think this change is OK?

        Show
        jingzhao Jing Zhao added a comment - The patch also makes the following changes in heartbeatCheck . - synchronized ( this ) { - dm.removeDeadDatanode(dead); - } + dm.removeDeadDatanode(dead); - synchronized ( this ) { - blockManager.removeBlocksAssociatedTo(failedStorage); - } + blockManager.removeBlocksAssociatedTo(failedStorage); Looks like the changes to heartbeat#datanodes are already protected by the heartbeat manager's monitor in corresponding methods. The heartbeat monitor was added originally by HDFS-2108 . Tsz Wo Nicholas Sze , do you think this change is OK?

          People

          • Assignee:
            jingzhao Jing Zhao
            Reporter:
            jingzhao Jing Zhao
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development