Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11674

reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recovered

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Scenario:
      1. 3 Node cluster with "dfs.client.block.write.replace-datanode-on-failure.policy" as DEFAULT
      Block is written with x data.
      2. One of the Datanode, NOT the first DN, is down
      3. Client tries to append data to block and fails since one DN is down.
      4. calls recoverLease() on the file.
      5. Successfull recovery happens.

      Issue:
      1. DNs which were connected from client before encountering mirror down, will have the reservedSpaceForReplicas incremented, BUT never decremented.
      2. So in long run DN's all space will be in reservedSpaceForReplicas resulting OutOfSpace errors.

      1. HDFS-11674-01.patch
        6 kB
        Vinayakumar B
      2. HDFS-11674-02.patch
        5 kB
        Vinayakumar B
      3. HDFS-11674-03.patch
        6 kB
        Vinayakumar B
      4. HDFS-11674-branch-2.7-03.patch
        7 kB
        Vinayakumar B

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.
        Hide
        vinayrpet Vinayakumar B added a comment -

        Thanks Arpit Agarwal and Brahma Reddy Battula for reviews.
        Committed to branch-2.7 as well.

        Show
        vinayrpet Vinayakumar B added a comment - Thanks Arpit Agarwal and Brahma Reddy Battula for reviews. Committed to branch-2.7 as well.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Vinayakumar B Nice Catch!! AS BR will not be closed in this scenario., bytes will not released forever.
        +1, LGTM on branch-2.7 too. Test failures are unrelated.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Vinayakumar B Nice Catch!! AS BR will not be closed in this scenario., bytes will not released forever. +1, LGTM on branch-2.7 too. Test failures are unrelated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 59s branch-2.7 passed
        +1 compile 1m 14s branch-2.7 passed with JDK v1.8.0_131
        +1 compile 1m 6s branch-2.7 passed with JDK v1.7.0_121
        +1 checkstyle 0m 28s branch-2.7 passed
        +1 mvnsite 1m 7s branch-2.7 passed
        +1 mvneclipse 0m 16s branch-2.7 passed
        +1 findbugs 3m 18s branch-2.7 passed
        +1 javadoc 1m 7s branch-2.7 passed with JDK v1.8.0_131
        +1 javadoc 1m 53s branch-2.7 passed with JDK v1.7.0_121
        +1 mvninstall 0m 56s the patch passed
        +1 compile 1m 4s the patch passed with JDK v1.8.0_131
        +1 javac 1m 4s the patch passed
        +1 compile 1m 4s the patch passed with JDK v1.7.0_121
        +1 javac 1m 4s the patch passed
        -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 245 unchanged - 2 fixed = 248 total (was 247)
        +1 mvnsite 1m 3s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        -1 whitespace 0m 0s The patch has 2141 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        -1 whitespace 0m 54s The patch 70 line(s) with tabs.
        +1 findbugs 3m 23s the patch passed
        +1 javadoc 1m 5s the patch passed with JDK v1.8.0_131
        +1 javadoc 1m 46s the patch passed with JDK v1.7.0_121
        -1 unit 47m 11s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        128m 18s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAAppend
          hadoop.hdfs.web.TestWebHdfsTokens
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
        JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.web.TestHttpsFileSystem
          hadoop.hdfs.server.namenode.TestCacheDirectives
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
        JDK v1.7.0_121 Timed out junit tests org.apache.hadoop.hdfs.web.TestWebHdfsTokens



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:c420dfe
        JIRA Issue HDFS-11674
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867706/HDFS-11674-branch-2.7-03.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux cd26782d1103 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / 28b9476
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19411/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19411/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19411/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19411/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19411/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19411/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 59s branch-2.7 passed +1 compile 1m 14s branch-2.7 passed with JDK v1.8.0_131 +1 compile 1m 6s branch-2.7 passed with JDK v1.7.0_121 +1 checkstyle 0m 28s branch-2.7 passed +1 mvnsite 1m 7s branch-2.7 passed +1 mvneclipse 0m 16s branch-2.7 passed +1 findbugs 3m 18s branch-2.7 passed +1 javadoc 1m 7s branch-2.7 passed with JDK v1.8.0_131 +1 javadoc 1m 53s branch-2.7 passed with JDK v1.7.0_121 +1 mvninstall 0m 56s the patch passed +1 compile 1m 4s the patch passed with JDK v1.8.0_131 +1 javac 1m 4s the patch passed +1 compile 1m 4s the patch passed with JDK v1.7.0_121 +1 javac 1m 4s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 245 unchanged - 2 fixed = 248 total (was 247) +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 15s the patch passed -1 whitespace 0m 0s The patch has 2141 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 54s The patch 70 line(s) with tabs. +1 findbugs 3m 23s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 46s the patch passed with JDK v1.7.0_121 -1 unit 47m 11s hadoop-hdfs in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 128m 18s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.web.TestHttpsFileSystem   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_121 Timed out junit tests org.apache.hadoop.hdfs.web.TestWebHdfsTokens Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HDFS-11674 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867706/HDFS-11674-branch-2.7-03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cd26782d1103 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 28b9476 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19411/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19411/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19411/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19411/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19411/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19411/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11728 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11728/)
        HDFS-11674. reserveSpaceForReplicas is not released if append request (vinayakumarb: rev 1411612aa4e70c704b941723217ed4efd8a0125b)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestSpaceReservation.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11728 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11728/ ) HDFS-11674 . reserveSpaceForReplicas is not released if append request (vinayakumarb: rev 1411612aa4e70c704b941723217ed4efd8a0125b) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestSpaceReservation.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
        Hide
        vinayrpet Vinayakumar B added a comment -

        Thanks Arpit Agarwal for confirmation.
        Committed to trunk, branch-2, branch-2.8, branch-2.8.1.

        Branch-2.7 have some conflicts, attaching patch for the same.

        Show
        vinayrpet Vinayakumar B added a comment - Thanks Arpit Agarwal for confirmation. Committed to trunk, branch-2, branch-2.8, branch-2.8.1. Branch-2.7 have some conflicts, attaching patch for the same.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        In the below part of code, blockLocations were queried first and then set as pipeline explicitly for the test purpose.

        That makes sense, thanks. I reran the test a few more times and didn't see another failure.

        +1 for the v3 patch.

        Show
        arpitagarwal Arpit Agarwal added a comment - In the below part of code, blockLocations were queried first and then set as pipeline explicitly for the test purpose. That makes sense, thanks. I reran the test a few more times and didn't see another failure. +1 for the v3 patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 13m 29s trunk passed
        +1 compile 0m 48s trunk passed
        +1 checkstyle 0m 37s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        -1 findbugs 1m 39s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 40s trunk passed
        +1 mvninstall 0m 47s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        +1 checkstyle 0m 35s the patch passed
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 43s the patch passed
        +1 javadoc 0m 37s the patch passed
        -1 unit 63m 6s hadoop-hdfs in the patch failed.
        -1 asflicense 0m 20s The patch generated 1 ASF License warnings.
        88m 44s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11674
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867499/HDFS-11674-03.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8b39078151c2 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 51b671e
        Default Java 1.8.0_121
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19396/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19396/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19396/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/19396/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19396/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 13m 29s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 14s trunk passed -1 findbugs 1m 39s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 43s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 63m 6s hadoop-hdfs in the patch failed. -1 asflicense 0m 20s The patch generated 1 ASF License warnings. 88m 44s Reason Tests Failed junit tests hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11674 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867499/HDFS-11674-03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8b39078151c2 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 51b671e Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19396/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/19396/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19396/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/19396/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19396/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vinayrpet Vinayakumar B added a comment -

        Attached the updated patch.

        There was a chance that test could timeout, where stopped datanode is chosen as primary for block recovery.
        Now marked the node as dead just before recovery in test.

        Show
        vinayrpet Vinayakumar B added a comment - Attached the updated patch. There was a chance that test could timeout, where stopped datanode is chosen as primary for block recovery. Now marked the node as dead just before recovery in test.
        Hide
        vinayrpet Vinayakumar B added a comment -

        Could you please clarify how this part works? getBlockLocations sorts the blocks by network distance from the caller, randomizing replicas at the same distance. So lastBlock.getLocations()[2] may be the first replica in the pipeline some times.

        In the below part of code, blockLocations were queried first and then set as pipeline explicitly for the test purpose. Also note that there is no 'sorting on distance' done for append calls. Its currently only for 'getBlockLocations()' cal. May be could do that in a following Jira.

        /*
         * Reset the pipeline for the append in such a way that, datanode which is
         * down is one of the mirror, not the first datanode.
         */
        HdfsBlockLocation blockLocation = (HdfsBlockLocation) fs.getClient()
            .getBlockLocations(file.toString(), 0, BLOCK_SIZE)[0];
        LocatedBlock lastBlock = blockLocation.getLocatedBlock();
        .
        .
        .
        DFSTestUtil.setPipeline((DFSOutputStream) os.getWrappedStream(),
          lastBlock);

        I ran this test 5 times and it timed out once waiting for the file to be closed. I didn't debug it further though.

        I will also check again, not sure whats wrong. But I am sure that its not because of current change or test. Could you paste console logs if possible.

        Show
        vinayrpet Vinayakumar B added a comment - Could you please clarify how this part works? getBlockLocations sorts the blocks by network distance from the caller, randomizing replicas at the same distance. So lastBlock.getLocations()[2] may be the first replica in the pipeline some times. In the below part of code, blockLocations were queried first and then set as pipeline explicitly for the test purpose. Also note that there is no 'sorting on distance' done for append calls. Its currently only for 'getBlockLocations()' cal. May be could do that in a following Jira. /* * Reset the pipeline for the append in such a way that, datanode which is * down is one of the mirror, not the first datanode. */ HdfsBlockLocation blockLocation = (HdfsBlockLocation) fs.getClient() .getBlockLocations(file.toString(), 0, BLOCK_SIZE)[0]; LocatedBlock lastBlock = blockLocation.getLocatedBlock(); . . . DFSTestUtil.setPipeline((DFSOutputStream) os.getWrappedStream(), lastBlock); I ran this test 5 times and it timed out once waiting for the file to be closed. I didn't debug it further though. I will also check again, not sure whats wrong. But I am sure that its not because of current change or test. Could you paste console logs if possible.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        I ran this test 5 times and it timed out once waiting for the file to be closed. I didn't debug it further though.

        "Thread-254"  prio=5 tid=465 runnable
        java.lang.Thread.State: RUNNABLE
                at java.lang.Thread.dumpThreads(Native Method)
                at java.lang.Thread.getAllStackTraces(Thread.java:1607)
                at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDump(TimedOutTestsListener.java:87)
                at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDiagnosticString(TimedOutTestsListener.java:73)
                at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:277)
                at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation.testReservedSpaceForLeaseRecovery(TestSpaceReservation.java:730)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
                at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
                at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
                at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
                at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
        "pool-56-thread-1" daemon prio=5 tid=596 timed_waiting
        
        Show
        arpitagarwal Arpit Agarwal added a comment - I ran this test 5 times and it timed out once waiting for the file to be closed. I didn't debug it further though. " Thread -254" prio=5 tid=465 runnable java.lang. Thread .State: RUNNABLE at java.lang. Thread .dumpThreads(Native Method) at java.lang. Thread .getAllStackTraces( Thread .java:1607) at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDump(TimedOutTestsListener.java:87) at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDiagnosticString(TimedOutTestsListener.java:73) at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:277) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation.testReservedSpaceForLeaseRecovery(TestSpaceReservation.java:730) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) "pool-56-thread-1" daemon prio=5 tid=596 timed_waiting
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        +1 for the patch. I am not clear on one thing in the test case:

            /*
             * Reset the pipeline for the append in such a way that, datanode which is
             * down is one of the mirror, not the first datanode.
             */
            HdfsBlockLocation blockLocation = (HdfsBlockLocation) fs.getClient()
                .getBlockLocations(file.toString(), 0, BLOCK_SIZE)[0];
            LocatedBlock lastBlock = blockLocation.getLocatedBlock();
            // stop 3rd node.
            cluster.stopDataNode(lastBlock.getLocations()[2].getName());
        

        Could you please clarify how this part works? getBlockLocations sorts the blocks by network distance from the caller, randomizing replicas at the same distance. So lastBlock.getLocations()[2] may be the first replica in the pipeline some times.

        Show
        arpitagarwal Arpit Agarwal added a comment - +1 for the patch. I am not clear on one thing in the test case: /* * Reset the pipeline for the append in such a way that, datanode which is * down is one of the mirror, not the first datanode. */ HdfsBlockLocation blockLocation = (HdfsBlockLocation) fs.getClient() .getBlockLocations(file.toString(), 0, BLOCK_SIZE)[0]; LocatedBlock lastBlock = blockLocation.getLocatedBlock(); // stop 3rd node. cluster.stopDataNode(lastBlock.getLocations()[2].getName()); Could you please clarify how this part works? getBlockLocations sorts the blocks by network distance from the caller, randomizing replicas at the same distance. So lastBlock.getLocations()[2] may be the first replica in the pipeline some times.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Hi Vinayakumar B, I'll take a look at it. Thanks for the heads up.

        Show
        arpitagarwal Arpit Agarwal added a comment - Hi Vinayakumar B , I'll take a look at it. Thanks for the heads up.
        Hide
        vinayrpet Vinayakumar B added a comment -

        Hi Arpit Agarwal, can you please review this?

        Show
        vinayrpet Vinayakumar B added a comment - Hi Arpit Agarwal , can you please review this?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 13m 40s trunk passed
        +1 compile 0m 59s trunk passed
        +1 checkstyle 0m 42s trunk passed
        +1 mvnsite 1m 2s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        -1 findbugs 1m 57s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 45s trunk passed
        +1 mvninstall 0m 50s the patch passed
        +1 compile 0m 47s the patch passed
        +1 javac 0m 47s the patch passed
        +1 checkstyle 0m 35s the patch passed
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 49s the patch passed
        +1 javadoc 0m 38s the patch passed
        -1 unit 71m 42s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        98m 41s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.TestStartup
          hadoop.hdfs.server.namenode.TestMetadataVersionOutput



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11674
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865878/HDFS-11674-02.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0df061d12f9e 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b0f54ea
        Default Java 1.8.0_121
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19260/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19260/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19260/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19260/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 13m 40s trunk passed +1 compile 0m 59s trunk passed +1 checkstyle 0m 42s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 15s trunk passed -1 findbugs 1m 57s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 45s trunk passed +1 mvninstall 0m 50s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 71m 42s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 98m 41s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.server.namenode.TestMetadataVersionOutput Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11674 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865878/HDFS-11674-02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0df061d12f9e 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b0f54ea Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19260/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/19260/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19260/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19260/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vinayrpet Vinayakumar B added a comment -

        Updated the patch.
        Please review.

        Show
        vinayrpet Vinayakumar B added a comment - Updated the patch. Please review.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 13m 16s trunk passed
        +1 compile 0m 47s trunk passed
        +1 checkstyle 0m 38s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 44s trunk passed
        +1 javadoc 0m 38s trunk passed
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 43s the patch passed
        +1 javac 0m 43s the patch passed
        -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 185 unchanged - 0 fixed = 187 total (was 185)
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 1m 51s the patch passed
        +1 javadoc 0m 37s the patch passed
        -1 unit 64m 42s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        90m 10s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestMaintenanceState
          hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ac17dc
        JIRA Issue HDFS-11674
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863960/HDFS-11674-01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 66d14fbf8867 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8c81a16
        Default Java 1.8.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19140/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19140/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19140/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19140/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19140/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 13m 16s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 38s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 185 unchanged - 0 fixed = 187 total (was 185) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 1m 51s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 64m 42s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 90m 10s Reason Tests Failed junit tests hadoop.hdfs.TestMaintenanceState   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HDFS-11674 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863960/HDFS-11674-01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 66d14fbf8867 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8c81a16 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19140/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19140/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19140/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19140/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19140/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vinayrpet Vinayakumar B added a comment -

        Attached the patch.

        Please review.

        Show
        vinayrpet Vinayakumar B added a comment - Attached the patch. Please review.

          People

          • Assignee:
            vinayrpet Vinayakumar B
            Reporter:
            vinayrpet Vinayakumar B
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development