Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: erasure-coding
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Below is the message taken from build:

      Error Message
      
      Time out waiting for EC block recovery.
      Stacktrace
      
      java.io.IOException: Time out waiting for EC block recovery.
      	at org.apache.hadoop.hdfs.TestRecoverStripedFile.waitForRecoveryFinished(TestRecoverStripedFile.java:383)
      	at org.apache.hadoop.hdfs.TestRecoverStripedFile.assertFileBlocksRecovery(TestRecoverStripedFile.java:283)
      	at org.apache.hadoop.hdfs.TestRecoverStripedFile.testRecoverAnyBlocks1(TestRecoverStripedFile.java:168)
      

      Reference : https://builds.apache.org/job/PreCommit-HDFS-Build/12758

      1. HDFS-9185-01.patch
        7 kB
        Rakesh R
      2. HDFS-9185-00.patch
        5 kB
        Rakesh R

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2386 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2386/)
        HDFS-9185. Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2386 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2386/ ) HDFS-9185 . Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #472 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/472/)
        HDFS-9185. Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413)

        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #472 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/472/ ) HDFS-9185 . Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #446 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/446/)
        HDFS-9185. Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #446 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/446/ ) HDFS-9185 . Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #480 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/480/)
        HDFS-9185. Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #480 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/480/ ) HDFS-9185 . Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1210 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1210/)
        HDFS-9185. Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413)

        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1210 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1210/ ) HDFS-9185 . Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2415 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2415/)
        HDFS-9185. Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2415 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2415/ ) HDFS-9185 . Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8555 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8555/)
        HDFS-9185. Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8555 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8555/ ) HDFS-9185 . Fix null tracer in ErasureCodingWorker. Contributed by Rakesh (jing9: rev c6cafc77e697317dad0708309b67b900a2e3a413) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
        Hide
        jingzhao Jing Zhao added a comment -

        I've committed this to trunk. Thanks for the contribution Rakesh R! Thanks for the review Uma Maheswara Rao G!

        Show
        jingzhao Jing Zhao added a comment - I've committed this to trunk. Thanks for the contribution Rakesh R ! Thanks for the review Uma Maheswara Rao G !
        Hide
        jingzhao Jing Zhao added a comment -

        The new patch looks good to me. All the failed tests passed in my local machine.

        +1. I will commit it shortly.

        Show
        jingzhao Jing Zhao added a comment - The new patch looks good to me. All the failed tests passed in my local machine. +1. I will commit it shortly.
        Hide
        rakeshr Rakesh R added a comment -

        Note: It looks like test case failures are not related to the patch. TestRecoverStripedFile case is consistently passing now.

        Show
        rakeshr Rakesh R added a comment - Note: It looks like test case failures are not related to the patch. TestRecoverStripedFile case is consistently passing now.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 19m 33s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 55s There were no new javac warning messages.
        +1 javadoc 10m 3s There were no new javadoc warning messages.
        -1 release audit 0m 15s The applied patch generated 1 release audit warnings.
        -1 checkstyle 2m 9s The applied patch generated 1 new checkstyle issues (total was 288, now 285).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 30s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 4m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 10s Pre-build of native portion
        -1 hdfs tests 181m 4s Tests failed in hadoop-hdfs.
        +1 hdfs tests 0m 30s Tests passed in hadoop-hdfs-client.
            232m 3s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.namenode.TestFSNamesystem
        Timed out tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
          org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
          org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache
          org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12764715/HDFS-9185-01.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / fd026f5
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12769/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12769/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12769/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12769/artifact/patchprocess/testrun_hadoop-hdfs.txt
        hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/12769/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12769/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12769/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 19m 33s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 55s There were no new javac warning messages. +1 javadoc 10m 3s There were no new javadoc warning messages. -1 release audit 0m 15s The applied patch generated 1 release audit warnings. -1 checkstyle 2m 9s The applied patch generated 1 new checkstyle issues (total was 288, now 285). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 10s Pre-build of native portion -1 hdfs tests 181m 4s Tests failed in hadoop-hdfs. +1 hdfs tests 0m 30s Tests passed in hadoop-hdfs-client.     232m 3s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestFSNamesystem Timed out tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics   org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement   org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache   org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12764715/HDFS-9185-01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fd026f5 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12769/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12769/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12769/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12769/artifact/patchprocess/testrun_hadoop-hdfs.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/12769/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12769/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12769/console This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Thank you Uma Maheswara Rao G, Jing Zhao for the review comments.

        Attached another patch addressing the above comment. Kindly review it again.

        Show
        rakeshr Rakesh R added a comment - Thank you Uma Maheswara Rao G , Jing Zhao for the review comments. Attached another patch addressing the above comment. Kindly review it again.
        Hide
        jingzhao Jing Zhao added a comment -

        Thanks for working on this, Rakesh R. The changes looks good to me. One comment is about the log level change. Changing the log level from debug to warn may generate unnecessary exception trace for DFSStripedInputStream since the failure can be covered by later decoding. So how about we change the log level for the unit test? We can need to add the following code to TestRecoverStripedBlocks:

        static {
          GenericTestUtils.setLogLevel(DFSClient.LOG, Level.ALL);
        }
        
        Show
        jingzhao Jing Zhao added a comment - Thanks for working on this, Rakesh R . The changes looks good to me. One comment is about the log level change. Changing the log level from debug to warn may generate unnecessary exception trace for DFSStripedInputStream since the failure can be covered by later decoding. So how about we change the log level for the unit test? We can need to add the following code to TestRecoverStripedBlocks : static { GenericTestUtils.setLogLevel(DFSClient.LOG, Level.ALL); }
        Hide
        rakeshr Rakesh R added a comment -

        Note: It seems test case failures are not related to the patch. Also, release audit and checkstyle warning are unrelated.

        Show
        rakeshr Rakesh R added a comment - Note: It seems test case failures are not related to the patch. Also, release audit and checkstyle warning are unrelated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 20m 12s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 8m 2s There were no new javac warning messages.
        +1 javadoc 9m 54s There were no new javadoc warning messages.
        -1 release audit 0m 15s The applied patch generated 1 release audit warnings.
        -1 checkstyle 2m 12s The applied patch generated 1 new checkstyle issues (total was 288, now 285).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 29s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 native 3m 9s Pre-build of native portion
        -1 hdfs tests 215m 47s Tests failed in hadoop-hdfs.
        +1 hdfs tests 0m 31s Tests passed in hadoop-hdfs-client.
            267m 17s  



        Reason Tests
        Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.util.TestByteArrayManager
          hadoop.hdfs.server.blockmanagement.TestNodeCount



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12764550/HDFS-9185-00.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 5db371f
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12760/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html
        Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12760/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12760/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
        hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12760/artifact/patchprocess/testrun_hadoop-hdfs.txt
        hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/12760/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12760/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12760/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 12s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 2s There were no new javac warning messages. +1 javadoc 9m 54s There were no new javadoc warning messages. -1 release audit 0m 15s The applied patch generated 1 release audit warnings. -1 checkstyle 2m 12s The applied patch generated 1 new checkstyle issues (total was 288, now 285). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 23s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 9s Pre-build of native portion -1 hdfs tests 215m 47s Tests failed in hadoop-hdfs. +1 hdfs tests 0m 31s Tests passed in hadoop-hdfs-client.     267m 17s   Reason Tests Failed unit tests hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.util.TestByteArrayManager   hadoop.hdfs.server.blockmanagement.TestNodeCount Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12764550/HDFS-9185-00.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5db371f Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12760/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html Release Audit https://builds.apache.org/job/PreCommit-HDFS-Build/12760/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12760/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12760/artifact/patchprocess/testrun_hadoop-hdfs.txt hadoop-hdfs-client test log https://builds.apache.org/job/PreCommit-HDFS-Build/12760/artifact/patchprocess/testrun_hadoop-hdfs-client.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12760/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12760/console This message was automatically generated.
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        Thank you Rakesh for reporting it. Changes looked good to me. Lets wait for jenkins to see this test failure fixed.

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - Thank you Rakesh for reporting it. Changes looked good to me. Lets wait for jenkins to see this test failure fixed.
        Hide
        rakeshr Rakesh R added a comment -

        Following is my analysis:

        1. ErasureCodingWorker is creating the RemoteBlockReader2 with null tracer, during the RemoteBlockReader2#read function call, it is hitting NPE and resulting in the failure. To fix this, how about passing the datanode#getTracer() to the reader ?
          ErasureCodingWorker .java
          
                  return RemoteBlockReader2.newBlockReader(
                      "dummy", block, blockToken, offsetInBlock, 
                      block.getNumBytes() - offsetInBlock, true,
                      "", newConnectedPeer(block, dnAddr, blockToken, dnInfo), dnInfo,
                      null, cachingStrategy, null);
          
          RemoteBlockReader2.java
          
            public synchronized int read(ByteBuffer buf) throws IOException {
              if (curDataSlice == null || curDataSlice.remaining() == 0 && bytesNeededToFinish > 0) {
                TraceScope scope = tracer.newScope(
                    "RemoteBlockReader2#readNextPacket(" + blockId + ")");
                try {
                  readNextPacket();
                } finally {
                  scope.close();
                }
              }
          
        2. The root cause is not visible in the log messages as StripedBlockUtil#getNextCompletedStripedRead() is logging the exception with DEBUG level, IMHO the log level has to be changed to INFO to know the failure reason.
          if (DFSClient.LOG.isDebugEnabled()) {
                  DFSClient.LOG.debug("ExecutionException " + e);
                }
          

        I'll soon prepare a patch including these changes.

        Show
        rakeshr Rakesh R added a comment - Following is my analysis: ErasureCodingWorker is creating the RemoteBlockReader2 with null tracer , during the RemoteBlockReader2#read function call, it is hitting NPE and resulting in the failure. To fix this, how about passing the datanode#getTracer() to the reader ? ErasureCodingWorker .java return RemoteBlockReader2.newBlockReader( "dummy" , block, blockToken, offsetInBlock, block.getNumBytes() - offsetInBlock, true , "", newConnectedPeer(block, dnAddr, blockToken, dnInfo), dnInfo, null , cachingStrategy, null ); RemoteBlockReader2.java public synchronized int read(ByteBuffer buf) throws IOException { if (curDataSlice == null || curDataSlice.remaining() == 0 && bytesNeededToFinish > 0) { TraceScope scope = tracer.newScope( "RemoteBlockReader2#readNextPacket(" + blockId + ")" ); try { readNextPacket(); } finally { scope.close(); } } The root cause is not visible in the log messages as StripedBlockUtil#getNextCompletedStripedRead() is logging the exception with DEBUG level, IMHO the log level has to be changed to INFO to know the failure reason. if (DFSClient.LOG.isDebugEnabled()) { DFSClient.LOG.debug( "ExecutionException " + e); } I'll soon prepare a patch including these changes.

          People

          • Assignee:
            rakeshr Rakesh R
            Reporter:
            rakeshr Rakesh R
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development