Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10824

MiniDFSCluster#storageCapacities has no effects on real capacity

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It has been noticed MiniDFSCluster#storageCapacities has no effects on real capacity. It can be reproduced by explicitly setting storageCapacities and then call ClientProtocol#getDatanodeStorageReport(DatanodeReportType.LIVE) to compare results. The following are storage report for one node with two volumes after I set capacity as 300 * 1024. Apparently, the capacity is not changed.

      adminState|DatanodeInfo$AdminStates (id=6861)

      blockPoolUsed 215192
      cacheCapacity 0
      cacheUsed 0
      capacity 998164971520
      datanodeUuid "839912e9-5bcb-45d1-81cf-9a9c9c02a00b" (id=6862)
      dependentHostNames LinkedList<E> (id=6863)
      dfsUsed 215192
      hostName "127.0.0.1" (id=6864)
      infoPort 64222
      infoSecurePort 0
      ipAddr "127.0.0.1" (id=6865)
      ipcPort 64223
      lastUpdate 1472682790948
      lastUpdateMonotonic 209605640
      level 0
      location "/default-rack" (id=6866)
      maintenanceExpireTimeInMS 0
      parent null
      peerHostName null
      remaining 20486512640
      softwareVersion null
      upgradeDomain null
      xceiverCount 1
      xferAddr "127.0.0.1:64220" (id=6855)
      xferPort 64220

      [0]StorageReport (id=6856)

      blockPoolUsed 4096
      capacity 499082485760
      dfsUsed 4096
      failed false
      remaining 10243256320
      storage DatanodeStorage (id=6869)

      [1]StorageReport (id=6859)

      blockPoolUsed 211096
      capacity 499082485760
      dfsUsed 211096
      failed false
      remaining 10243256320
      storage DatanodeStorage (id=6872)
      1. HDFS-10824-branch-2.006.patch
        11 kB
        Xiaobing Zhou
      2. HDFS-10824.006.patch
        11 kB
        Xiaobing Zhou
      3. HDFS-10824.005.patch
        11 kB
        Xiaobing Zhou
      4. HDFS-10824.004.patch
        11 kB
        Xiaobing Zhou
      5. HDFS-10824.003.patch
        10 kB
        Xiaobing Zhou
      6. HDFS-10824.002.patch
        10 kB
        Xiaobing Zhou
      7. HDFS-10824.001.patch
        12 kB
        Xiaobing Zhou
      8. HDFS-10824.000.patch
        11 kB
        Xiaobing Zhou

        Activity

        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thanks for the branch-2 patch Xiaobing Zhou. Committed this for 2.8.0.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thanks for the branch-2 patch Xiaobing Zhou . Committed this for 2.8.0.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 40s branch-2 passed
        +1 compile 0m 42s branch-2 passed with JDK v1.8.0_101
        +1 compile 0m 42s branch-2 passed with JDK v1.7.0_111
        +1 checkstyle 0m 29s branch-2 passed
        +1 mvnsite 0m 51s branch-2 passed
        +1 mvneclipse 0m 16s branch-2 passed
        +1 findbugs 1m 55s branch-2 passed
        +1 javadoc 0m 55s branch-2 passed with JDK v1.8.0_101
        +1 javadoc 1m 34s branch-2 passed with JDK v1.7.0_111
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 37s the patch passed with JDK v1.8.0_101
        +1 javac 0m 37s the patch passed
        +1 compile 0m 40s the patch passed with JDK v1.7.0_111
        +1 javac 0m 40s the patch passed
        -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 189 unchanged - 2 fixed = 190 total (was 191)
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 9s the patch passed
        +1 javadoc 0m 53s the patch passed with JDK v1.8.0_101
        +1 javadoc 1m 34s the patch passed with JDK v1.7.0_111
        -1 unit 47m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        125m 40s



        Reason Tests
        JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAMetrics
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
        JDK v1.7.0_111 Failed junit tests hadoop.hdfs.TestDFSShell



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Issue HDFS-10824
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830781/HDFS-10824-branch-2.006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3a771e888331 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / baf8aac
        Default Java 1.7.0_111
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16915/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16915/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
        JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16915/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16915/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 40s branch-2 passed +1 compile 0m 42s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 42s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 0m 29s branch-2 passed +1 mvnsite 0m 51s branch-2 passed +1 mvneclipse 0m 16s branch-2 passed +1 findbugs 1m 55s branch-2 passed +1 javadoc 0m 55s branch-2 passed with JDK v1.8.0_101 +1 javadoc 1m 34s branch-2 passed with JDK v1.7.0_111 +1 mvninstall 0m 45s the patch passed +1 compile 0m 37s the patch passed with JDK v1.8.0_101 +1 javac 0m 37s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_111 +1 javac 0m 40s the patch passed -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 189 unchanged - 2 fixed = 190 total (was 191) +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 0m 53s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 34s the patch passed with JDK v1.7.0_111 -1 unit 47m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_111. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 125m 40s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAMetrics   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA JDK v1.7.0_111 Failed junit tests hadoop.hdfs.TestDFSShell Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-10824 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830781/HDFS-10824-branch-2.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3a771e888331 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / baf8aac Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16915/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16915/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16915/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16915/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        Thank you Arpit Agarwal, Chris Nauroth and Anu Engineer for reviewing/committing it. I posted branch-2 patch and re-opened the Jira to kick off Jenkins run.

        Show
        xiaobingo Xiaobing Zhou added a comment - Thank you Arpit Agarwal , Chris Nauroth and Anu Engineer for reviewing/committing it. I posted branch-2 patch and re-opened the Jira to kick off Jenkins run.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10508 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10508/)
        HDFS-10824. MiniDFSCluster#storageCapacities has no effects on real (arp: rev c3b235e56597d55387b4003e376faee10b473d55)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMiniDFSCluster.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10508 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10508/ ) HDFS-10824 . MiniDFSCluster#storageCapacities has no effects on real (arp: rev c3b235e56597d55387b4003e376faee10b473d55) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMiniDFSCluster.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        +1 I committed this to trunk. Thanks Xiaobing Zhou and thanks Anu Engineer and Chris Nauroth for the reviews.

        Xiaobing, if you want to post a branch-2 patch I can commit that too. The conflict looks straightforward but I'd prefer Jenkins do a full unit test run to be safe.

        Show
        arpitagarwal Arpit Agarwal added a comment - +1 I committed this to trunk. Thanks Xiaobing Zhou and thanks Anu Engineer and Chris Nauroth for the reviews. Xiaobing, if you want to post a branch-2 patch I can commit that too. The conflict looks straightforward but I'd prefer Jenkins do a full unit test run to be safe.
        Hide
        anu Anu Engineer added a comment -

        I am good with the changes. Thanks Chris Nauroth and Arpit Agarwal for the code reviews and Xiaobing Zhou for the patch.

        Show
        anu Anu Engineer added a comment - I am good with the changes. Thanks Chris Nauroth and Arpit Agarwal for the code reviews and Xiaobing Zhou for the patch.
        Hide
        cnauroth Chris Nauroth added a comment -

        +1 for patch 006. Thank you, Xiaobing.

        Anu Engineer or Arpit Agarwal, do you have any further comments?

        Show
        cnauroth Chris Nauroth added a comment - +1 for patch 006. Thank you, Xiaobing. Anu Engineer or Arpit Agarwal , do you have any further comments?
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        It seems that failures are note related the patch. They passed in trunk w/ and w/o the patch v006.

        Show
        xiaobingo Xiaobing Zhou added a comment - It seems that failures are note related the patch. They passed in trunk w/ and w/o the patch v006.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 52s trunk passed
        +1 compile 0m 44s trunk passed
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 0m 50s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 39s trunk passed
        +1 javadoc 0m 55s trunk passed
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 41s the patch passed
        +1 javac 0m 41s the patch passed
        -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 205 unchanged - 1 fixed = 206 total (was 206)
        +1 mvnsite 0m 47s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 45s the patch passed
        +1 javadoc 0m 53s the patch passed
        -1 unit 65m 17s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        84m 17s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestErasureCodeBenchmarkThroughput
          hadoop.hdfs.TestRenameWhileOpen



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10824
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830537/HDFS-10824.006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 9ef277460c58 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8ae4729
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16882/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16882/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16882/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16882/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 52s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 205 unchanged - 1 fixed = 206 total (was 206) +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 53s the patch passed -1 unit 65m 17s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 84m 17s Reason Tests Failed junit tests hadoop.hdfs.TestErasureCodeBenchmarkThroughput   hadoop.hdfs.TestRenameWhileOpen Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10824 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830537/HDFS-10824.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9ef277460c58 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8ae4729 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16882/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16882/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16882/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16882/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        Chris Nauroth, you are correct, we should add the hear beat. Thanks. v006 is posted.

        Show
        xiaobingo Xiaobing Zhou added a comment - Chris Nauroth , you are correct, we should add the hear beat. Thanks. v006 is posted.
        Hide
        cnauroth Chris Nauroth added a comment -

        Xiaobing Zhou, thank you for the update.

        I thought of one more thing. I think it's still necessary to send a heartbeat, so that the NameNode updates its view of the capacity numbers before any test assertions about capacity seen at the NameNode. I think it would be safe to add back a call to DataNodeTestUtils#triggerHeartbeat right after the call to FsVolumeImpl#setCapacityForTesting, because you are now waiting for the DataNode to be fully initialized before that logic runs.

        Show
        cnauroth Chris Nauroth added a comment - Xiaobing Zhou , thank you for the update. I thought of one more thing. I think it's still necessary to send a heartbeat, so that the NameNode updates its view of the capacity numbers before any test assertions about capacity seen at the NameNode. I think it would be safe to add back a call to DataNodeTestUtils#triggerHeartbeat right after the call to FsVolumeImpl#setCapacityForTesting , because you are now waiting for the DataNode to be fully initialized before that logic runs.
        Hide
        xiaobingo Xiaobing Zhou added a comment - - edited

        The test failure is not related to the patch. It actually passed with latest code in my local. It's been reported by HDFS-10426.

        Show
        xiaobingo Xiaobing Zhou added a comment - - edited The test failure is not related to the patch. It actually passed with latest code in my local. It's been reported by HDFS-10426 .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 7m 28s trunk passed
        +1 compile 0m 48s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 54s trunk passed
        +1 javadoc 1m 5s trunk passed
        +1 mvninstall 0m 58s the patch passed
        +1 compile 0m 55s the patch passed
        +1 javac 0m 55s the patch passed
        -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 205 unchanged - 1 fixed = 206 total (was 206)
        +1 mvnsite 0m 58s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 6s the patch passed
        +1 javadoc 1m 4s the patch passed
        -1 unit 67m 44s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        89m 36s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10824
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830427/HDFS-10824.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 76504fbe1d1c 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 059058f
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16873/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16873/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16873/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16873/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 28s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 1m 5s trunk passed +1 mvninstall 0m 58s the patch passed +1 compile 0m 55s the patch passed +1 javac 0m 55s the patch passed -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 205 unchanged - 1 fixed = 206 total (was 206) +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 1m 4s the patch passed -1 unit 67m 44s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 89m 36s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10824 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830427/HDFS-10824.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 76504fbe1d1c 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 059058f Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16873/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16873/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16873/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16873/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        Thanks Chris Nauroth for review. v005 is posted by moving waitDataNodeFullyStarted.

        Show
        xiaobingo Xiaobing Zhou added a comment - Thanks Chris Nauroth for review. v005 is posted by moving waitDataNodeFullyStarted.
        Hide
        cnauroth Chris Nauroth added a comment -

        Xiaobing Zhou, thank you for sharing patch 004.

        Adding the waitDataNodeFullyStarted call for all DataNode restarts might slow down some tests that don't really need the wait. Do you think that call can be moved inside setDataNodeStorageCapacities? That way, you could do the wait only if storageCapacities is non-null and non-empty.

        Show
        cnauroth Chris Nauroth added a comment - Xiaobing Zhou , thank you for sharing patch 004. Adding the waitDataNodeFullyStarted call for all DataNode restarts might slow down some tests that don't really need the wait. Do you think that call can be moved inside setDataNodeStorageCapacities ? That way, you could do the wait only if storageCapacities is non-null and non-empty.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 44s trunk passed
        +1 compile 0m 45s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 41s trunk passed
        +1 javadoc 0m 55s trunk passed
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 41s the patch passed
        +1 javac 0m 41s the patch passed
        -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 205 unchanged - 1 fixed = 206 total (was 206)
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 46s the patch passed
        +1 javadoc 0m 52s the patch passed
        +1 unit 58m 6s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        76m 53s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10824
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829442/HDFS-10824.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5ba6a4674d51 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e80386d
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16814/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16814/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16814/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 44s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 205 unchanged - 1 fixed = 206 total (was 206) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 52s the patch passed +1 unit 58m 6s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 76m 53s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10824 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829442/HDFS-10824.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5ba6a4674d51 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e80386d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16814/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16814/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16814/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        Posted v004. Waiting for data node fully started is correct way for syncing storage initialization.

        Show
        xiaobingo Xiaobing Zhou added a comment - Posted v004. Waiting for data node fully started is correct way for syncing storage initialization.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        Thank you Chris Nauroth and Arpit Agarwal for comments. I will look into the failure.
        storageCap is intentionally typed as List<long[]> since startDataNodes is used to start new DNs in existing cluster. The list is to retain old cap setting and append new caps so that they are both remembered for DNs restart.

        Show
        xiaobingo Xiaobing Zhou added a comment - Thank you Chris Nauroth and Arpit Agarwal for comments. I will look into the failure. storageCap is intentionally typed as List<long[]> since startDataNodes is used to start new DNs in existing cluster. The list is to retain old cap setting and append new caps so that they are both remembered for DNs restart.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thanks for fixing this Xiaobing Zhou. The test failure pointed out by Chris Nauroth also repro'd for me.

        Minor comment - the member storageCap can just be a long[][] to avoid the conversions on lines 1663 and 2303.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thanks for fixing this Xiaobing Zhou . The test failure pointed out by Chris Nauroth also repro'd for me. Minor comment - the member storageCap can just be a long[][] to avoid the conversions on lines 1663 and 2303.
        Hide
        cnauroth Chris Nauroth added a comment -

        Xiaobing Zhou, thank you for the patch.

        It appears that at least one of the test failures, TestFsDatasetImpl, was caused by patch revision 003. That test passes for me on current trunk, and then it times out after I apply patch 003.

        I didn't fully investigate root cause. However, I did run jstack on the JUnit process to see what was happening. I've pasted the relevant stack trace for the main thread below. After restarting the mini-cluster, the thread is blocked while trying to trigger a heartbeat. Perhaps something in the patch has impacted reinitialization after DataNode restart, such as delivery of the initial block report.

        I hope this helps with investigation.

        "main" #1 prio=5 os_prio=31 tid=0x00007fee83801800 nid=0x1703 in Object.wait() [0x0000700000218000]
           java.lang.Thread.State: TIMED_WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.triggerHeartbeatForTests(BPServiceActor.java:310)
        	- locked <0x000000079ae302c0> (a org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager)
        	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.triggerHeartbeatForTests(BPOfferService.java:592)
        	at org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils.triggerHeartbeat(DataNodeTestUtils.java:72)
        	at org.apache.hadoop.hdfs.MiniDFSCluster.restartDataNode(MiniDFSCluster.java:2289)
        	- locked <0x00000007400415e0> (a org.apache.hadoop.hdfs.MiniDFSCluster)
        	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testAddVolumeWithSameStorageUuid(TestFsDatasetImpl.java:242)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:498)
        	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
        	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
        	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
        	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
        
        Show
        cnauroth Chris Nauroth added a comment - Xiaobing Zhou , thank you for the patch. It appears that at least one of the test failures, TestFsDatasetImpl , was caused by patch revision 003. That test passes for me on current trunk, and then it times out after I apply patch 003. I didn't fully investigate root cause. However, I did run jstack on the JUnit process to see what was happening. I've pasted the relevant stack trace for the main thread below. After restarting the mini-cluster, the thread is blocked while trying to trigger a heartbeat. Perhaps something in the patch has impacted reinitialization after DataNode restart, such as delivery of the initial block report. I hope this helps with investigation. "main" #1 prio=5 os_prio=31 tid=0x00007fee83801800 nid=0x1703 in Object .wait() [0x0000700000218000] java.lang. Thread .State: TIMED_WAITING (on object monitor) at java.lang. Object .wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.triggerHeartbeatForTests(BPServiceActor.java:310) - locked <0x000000079ae302c0> (a org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.triggerHeartbeatForTests(BPOfferService.java:592) at org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils.triggerHeartbeat(DataNodeTestUtils.java:72) at org.apache.hadoop.hdfs.MiniDFSCluster.restartDataNode(MiniDFSCluster.java:2289) - locked <0x00000007400415e0> (a org.apache.hadoop.hdfs.MiniDFSCluster) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testAddVolumeWithSameStorageUuid(TestFsDatasetImpl.java:242) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        The test failure is not related to the patch.

        Show
        xiaobingo Xiaobing Zhou added a comment - The test failure is not related to the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 8m 28s trunk passed
        +1 compile 0m 47s trunk passed
        +1 checkstyle 0m 31s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 51s trunk passed
        +1 javadoc 0m 59s trunk passed
        +1 mvninstall 0m 54s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 205 unchanged - 1 fixed = 206 total (was 206)
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 49s the patch passed
        +1 javadoc 0m 57s the patch passed
        -1 unit 83m 11s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        104m 53s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
        Timed out junit tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10824
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828715/HDFS-10824.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b2cd39117965 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / fcbac00
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16755/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16755/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16755/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16755/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 28s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 51s trunk passed +1 javadoc 0m 59s trunk passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 205 unchanged - 1 fixed = 206 total (was 206) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 57s the patch passed -1 unit 83m 11s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 104m 53s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency Timed out junit tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10824 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828715/HDFS-10824.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b2cd39117965 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fcbac00 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16755/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16755/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16755/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16755/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        v003 fixed test failures.

        Show
        xiaobingo Xiaobing Zhou added a comment - v003 fixed test failures.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 7m 22s trunk passed
        +1 compile 0m 55s trunk passed
        +1 checkstyle 0m 31s trunk passed
        +1 mvnsite 1m 7s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 59s trunk passed
        +1 javadoc 1m 2s trunk passed
        +1 mvninstall 1m 5s the patch passed
        +1 compile 0m 44s the patch passed
        +1 javac 0m 44s the patch passed
        -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 205 unchanged - 1 fixed = 206 total (was 206)
        +1 mvnsite 0m 58s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 0s the patch passed
        +1 javadoc 0m 58s the patch passed
        -1 unit 83m 16s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        105m 13s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestRead
          hadoop.hdfs.TestDFSStorageStateRecovery
          hadoop.hdfs.TestPread
          hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
          hadoop.hdfs.TestDFSStripedInputStream
          hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
          hadoop.hdfs.server.datanode.TestDataNodeTransferSocketSize
          hadoop.hdfs.TestDFSRollback
          hadoop.hdfs.TestDFSUpgrade
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
          hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
          hadoop.hdfs.TestSetrepIncreasing
          hadoop.hdfs.TestRenameWhileOpen
          hadoop.hdfs.server.datanode.TestDataNodeMetrics
          hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
          hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
          hadoop.hdfs.TestInjectionForSimulatedStorage
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          hadoop.hdfs.TestDecommission
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.server.datanode.TestDataNodeInitStorage
          hadoop.hdfs.TestFileCreation
          hadoop.hdfs.TestWriteBlockGetsBlockLengthHint
          hadoop.hdfs.server.namenode.TestFileLimit
          hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
          hadoop.hdfs.TestSmallBlock
          hadoop.hdfs.TestReplication
          hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
        Timed out junit tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10824
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828534/HDFS-10824.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 45b7193e7d55 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2a8f55a
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16742/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16742/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16742/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16742/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 22s trunk passed +1 compile 0m 55s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 1m 2s trunk passed +1 mvninstall 1m 5s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 205 unchanged - 1 fixed = 206 total (was 206) +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 0m 58s the patch passed -1 unit 83m 16s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 105m 13s Reason Tests Failed junit tests hadoop.hdfs.TestRead   hadoop.hdfs.TestDFSStorageStateRecovery   hadoop.hdfs.TestPread   hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.TestDFSStripedInputStream   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.datanode.TestDataNodeTransferSocketSize   hadoop.hdfs.TestDFSRollback   hadoop.hdfs.TestDFSUpgrade   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration   hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer   hadoop.hdfs.TestSetrepIncreasing   hadoop.hdfs.TestRenameWhileOpen   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer   hadoop.hdfs.TestInjectionForSimulatedStorage   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestDecommission   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.datanode.TestDataNodeInitStorage   hadoop.hdfs.TestFileCreation   hadoop.hdfs.TestWriteBlockGetsBlockLengthHint   hadoop.hdfs.server.namenode.TestFileLimit   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.TestSmallBlock   hadoop.hdfs.TestReplication   hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks Timed out junit tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10824 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828534/HDFS-10824.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 45b7193e7d55 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2a8f55a Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16742/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16742/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16742/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16742/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment - - edited

        Thanks for review Anu Engineer. v002 is posted.
        1. The member is named as storageCap to avoid edits. storageCapacities in function startDataNodes is intended for starting additional DNs. so memorizing capacity is changed accordingly.
        2. It's better not to remove storageCapacities parameters, since startDataNodes is designed to start additional DNs in on-going cluster by providing diff capacities.
        3. tiggerHeartbeat is to wait for local DN storage to be initialized after block pool has successfully connected to its NN. See also DataNode#runDatanodeDaemon -> blockPoolManager.startAll() --> BPOfferService.start --> BPServiceActor.start --> BPServiceActor.run -->BPServiceActor.connectToNNAndHandshake, storage initialization is triggered async. tiggerHeartbeat is necessary in this case, although triggerBlock not.
        4. passed different capacities.

        Show
        xiaobingo Xiaobing Zhou added a comment - - edited Thanks for review Anu Engineer . v002 is posted. 1. The member is named as storageCap to avoid edits. storageCapacities in function startDataNodes is intended for starting additional DNs. so memorizing capacity is changed accordingly. 2. It's better not to remove storageCapacities parameters, since startDataNodes is designed to start additional DNs in on-going cluster by providing diff capacities. 3. tiggerHeartbeat is to wait for local DN storage to be initialized after block pool has successfully connected to its NN. See also DataNode#runDatanodeDaemon -> blockPoolManager.startAll() --> BPOfferService.start --> BPServiceActor.start --> BPServiceActor.run -->BPServiceActor.connectToNNAndHandshake, storage initialization is triggered async. tiggerHeartbeat is necessary in this case, although triggerBlock not. 4. passed different capacities.
        Hide
        anu Anu Engineer added a comment - - edited

        Thanks for the patch. Couple of Questions.

        1. I was wondering if storageCapacities is a good member name at all ? if you had called it StorageCap instead of storageCapacities, you save lots of unnecessary edits where you are replacing the storageCapacities with StorageCap.
        2. Since we made storageCapacities a member, does it make sense it pass thru as an argument to all functions. Most of the time, storageCap is nothing but this.StorageCapacities. So was wondering if we should just use the member variable directly ?
        3. Could you please tell me why we have added tiggerHeartbeat and triggerBlock in line 2288 and 2289. I am not able to understand why we need to trigger them manually with datanode restart.
        4. In the test, can we please pass different capacities to MiniDFSCluster and make sure that we are able to support that ?
        Show
        anu Anu Engineer added a comment - - edited Thanks for the patch. Couple of Questions. I was wondering if storageCapacities is a good member name at all ? if you had called it StorageCap instead of storageCapacities, you save lots of unnecessary edits where you are replacing the storageCapacities with StorageCap. Since we made storageCapacities a member, does it make sense it pass thru as an argument to all functions. Most of the time, storageCap is nothing but this.StorageCapacities . So was wondering if we should just use the member variable directly ? Could you please tell me why we have added tiggerHeartbeat and triggerBlock in line 2288 and 2289. I am not able to understand why we need to trigger them manually with datanode restart. In the test, can we please pass different capacities to MiniDFSCluster and make sure that we are able to support that ?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 58s trunk passed
        +1 compile 0m 43s trunk passed
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 42s trunk passed
        +1 javadoc 0m 54s trunk passed
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 41s the patch passed
        +1 javac 0m 41s the patch passed
        -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 200 unchanged - 6 fixed = 205 total (was 206)
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 53s the patch passed
        -1 unit 66m 4s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        85m 4s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestSafeMode
          hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
        Timed out junit tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10824
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828147/HDFS-10824.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e7bd3e626e59 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 72dfb04
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16724/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16724/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16724/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16724/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 58s trunk passed +1 compile 0m 43s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 200 unchanged - 6 fixed = 205 total (was 206) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 53s the patch passed -1 unit 66m 4s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 85m 4s Reason Tests Failed junit tests hadoop.hdfs.TestSafeMode   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages Timed out junit tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10824 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828147/HDFS-10824.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e7bd3e626e59 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 72dfb04 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16724/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16724/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16724/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16724/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment - - edited

        v001 is posted. It fixed test failures and check style issues.

        Show
        xiaobingo Xiaobing Zhou added a comment - - edited v001 is posted. It fixed test failures and check style issues.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 58s trunk passed
        +1 compile 0m 45s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 44s trunk passed
        +1 javadoc 0m 55s trunk passed
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 205 unchanged - 1 fixed = 213 total (was 206)
        +1 mvnsite 0m 51s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 1s The patch has no whitespace issues.
        +1 findbugs 1m 50s the patch passed
        +1 javadoc 0m 52s the patch passed
        -1 unit 66m 19s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        85m 39s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestEncryptedTransfer
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.TestClientProtocolForPipelineRecovery
          hadoop.hdfs.TestPread
          hadoop.hdfs.TestDecommission
          hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC
          hadoop.hdfs.server.blockmanagement.TestNodeCount
          hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
          hadoop.hdfs.TestDatanodeRegistration
          hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
          hadoop.net.TestNetworkTopology
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
          hadoop.hdfs.TestFileChecksum
          hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.TestLeaseRecovery
          hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages
          hadoop.hdfs.server.datanode.TestDataNodeMXBean
          hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
          hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          hadoop.hdfs.server.namenode.ha.TestHASafeMode
          hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
          hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
          hadoop.hdfs.server.datanode.TestDataNodeUUID
          hadoop.hdfs.server.diskbalancer.TestDiskBalancerWithMockMover
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.TestDFSClientExcludedNodes
          hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
          hadoop.hdfs.TestSafeMode
          hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
          hadoop.hdfs.TestSafeModeWithStripedFile
          hadoop.hdfs.TestMaintenanceState
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestReplication
          hadoop.hdfs.TestDecommissionWithStriped
          hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
          hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
          hadoop.hdfs.TestDataTransferKeepalive
        Timed out junit tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10824
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828117/HDFS-10824.000.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 13343d93502b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 58ed4fa
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16717/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/16717/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16717/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16717/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 58s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 205 unchanged - 1 fixed = 213 total (was 206) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 findbugs 1m 50s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 66m 19s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 85m 39s Reason Tests Failed junit tests hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestPread   hadoop.hdfs.TestDecommission   hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade   hadoop.hdfs.TestDatanodeRegistration   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.net.TestNetworkTopology   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   hadoop.hdfs.TestFileChecksum   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.TestLeaseRecovery   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages   hadoop.hdfs.server.datanode.TestDataNodeMXBean   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.server.namenode.ha.TestHASafeMode   hadoop.hdfs.server.namenode.ha.TestStandbyIsHot   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.server.namenode.TestProcessCorruptBlocks   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.diskbalancer.TestDiskBalancerWithMockMover   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestDFSClientExcludedNodes   hadoop.hdfs.server.namenode.TestListCorruptFileBlocks   hadoop.hdfs.TestSafeMode   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.TestMaintenanceState   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestReplication   hadoop.hdfs.TestDecommissionWithStriped   hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness   hadoop.hdfs.TestDataTransferKeepalive Timed out junit tests org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10824 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828117/HDFS-10824.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 13343d93502b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 58ed4fa Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16717/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16717/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16717/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16717/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment - - edited

        v000 is posted for review. The idea is to memorize storage capacity setting which is used for resetting after restart.

        Show
        xiaobingo Xiaobing Zhou added a comment - - edited v000 is posted for review. The idea is to memorize storage capacity setting which is used for resetting after restart.

          People

          • Assignee:
            xiaobingo Xiaobing Zhou
            Reporter:
            xiaobingo Xiaobing Zhou
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development