Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7916

'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      if any badblock found, then BPSA for StandbyNode will go for infinite times to report it.

      2015-03-11 19:43:41,528 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Failed to report bad block BP-1384821822-10.224.54.68-1422634566395:blk_1079544278_5812006 to namenode: stobdtserver3/10.224.54.70:18010
      org.apache.hadoop.hdfs.server.datanode.BPServiceActorActionException: Failed to report bad block BP-1384821822-10.224.54.68-1422634566395:blk_1079544278_5812006 to namenode:
              at org.apache.hadoop.hdfs.server.datanode.ReportBadBlockAction.reportTo(ReportBadBlockAction.java:63)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processQueueMessages(BPServiceActor.java:1020)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:762)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:856)
              at java.lang.Thread.run(Thread.java:745)
      
      1. HDFS-7916-01.patch
        2 kB
        Vinayakumar B
      2. HDFS-7916-1.patch
        6 kB
        Rushabh S Shah

        Issue Links

          Activity

          Hide
          vinayrpet Vinayakumar B added a comment -

          BPServiceActor#processQueueMessages() tries to execute the ReportBadBlockAction#reportTo(..) and on any exception, it will add back to queue.
          So in case of standby node, adding back to queue should be ignored for the bad block.

          Show
          vinayrpet Vinayakumar B added a comment - BPServiceActor#processQueueMessages() tries to execute the ReportBadBlockAction#reportTo(..) and on any exception, it will add back to queue. So in case of standby node, adding back to queue should be ignored for the bad block.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Attaching the patch for the same.

          Show
          vinayrpet Vinayakumar B added a comment - Attaching the patch for the same.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Hi Vinayakumar B,
          I implemented this feature in HDFS-7704.
          I missed out this case.
          apologies for the inconvenience caused.
          The patch looks good to me.
          +1 non binding.

          Show
          shahrs87 Rushabh S Shah added a comment - Hi Vinayakumar B , I implemented this feature in HDFS-7704 . I missed out this case. apologies for the inconvenience caused. The patch looks good to me. +1 non binding.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12703880/HDFS-7916-01.patch
          against trunk revision 30c428a.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager
          org.apache.hadoop.hdfs.TestAppendSnapshotTruncate

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9833//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9833//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703880/HDFS-7916-01.patch against trunk revision 30c428a. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager org.apache.hadoop.hdfs.TestAppendSnapshotTruncate Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9833//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9833//console This message is automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          latest patch attached is just to solve the jenkins problem. Not related to this issue,

          Show
          vinayrpet Vinayakumar B added a comment - latest patch attached is just to solve the jenkins problem. Not related to this issue,
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12705022/HDFS-7916-02.patch
          against trunk revision 7179f94.

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9923//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12705022/HDFS-7916-02.patch against trunk revision 7179f94. -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9923//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12705101/HDFS-7916-002.patch
          against trunk revision 7179f94.

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9924//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12705101/HDFS-7916-002.patch against trunk revision 7179f94. -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9924//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12705103/HDFS-7916-002.patch
          against trunk revision 7179f94.

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9925//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12705103/HDFS-7916-002.patch against trunk revision 7179f94. -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9925//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12705103/HDFS-7916-002.patch
          against trunk revision a89b087.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9927//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12705103/HDFS-7916-002.patch against trunk revision a89b087. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9927//console This message is automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          Thanks Vinay for the patch, it looks good, one comment, +1 after addressing:
          Please do the same handle in ErrorReportAction.

          Show
          hitliuyi Yi Liu added a comment - Thanks Vinay for the patch, it looks good, one comment, +1 after addressing: Please do the same handle in ErrorReportAction .
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Yi Liu: This standby exception is thrown by checkOperation(OperationCategory.WRITE) function in FSNamesystem#reportBadBlocks.
          This check is not there for NameNodeRpcServer#errorReport(). So I think there is no need to handle StandByException in ErrorReportAction.
          Please correct me if I am wrong.

          Show
          shahrs87 Rushabh S Shah added a comment - Yi Liu : This standby exception is thrown by checkOperation(OperationCategory.WRITE) function in FSNamesystem#reportBadBlocks. This check is not there for NameNodeRpcServer#errorReport(). So I think there is no need to handle StandByException in ErrorReportAction. Please correct me if I am wrong.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Yes. Rushabh S Shah is right. I had already checked about this. ErrorReportAction doesn't need to be handled.

          Show
          vinayrpet Vinayakumar B added a comment - Yes. Rushabh S Shah is right. I had already checked about this. ErrorReportAction doesn't need to be handled.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12703880/HDFS-7916-01.patch
          against trunk revision 2e79f1c.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10141//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10141//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703880/HDFS-7916-01.patch against trunk revision 2e79f1c. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10141//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10141//console This message is automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          +1

          Show
          hitliuyi Yi Liu added a comment - +1
          Hide
          vinayrpet Vinayakumar B added a comment -

          Committed to trunk and branch-2.
          Thanks Yi Liu and Rushabh S Shah.

          Show
          vinayrpet Vinayakumar B added a comment - Committed to trunk and branch-2. Thanks Yi Liu and Rushabh S Shah .
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7492 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7492/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7492 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7492/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #885 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/885/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #885 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/885/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #151 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/151/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #151 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/151/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2083 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2083/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2083 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2083/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #142 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/142/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #142 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/142/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #151 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/151/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #151 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/151/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2101 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2101/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2101 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2101/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 867d5d2675b8fb73c40fac1e581b02b005459d95) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          kihwal Kihwal Lee added a comment -

          Vinayakumar B, I think we need it for 2.7 (2.7.1?).

          Show
          kihwal Kihwal Lee added a comment - Vinayakumar B , I think we need it for 2.7 (2.7.1?).
          Hide
          vinayrpet Vinayakumar B added a comment -

          Thanks Kihwal Lee for the reminder.
          I will merge it to 2.7.1 soon.

          Show
          vinayrpet Vinayakumar B added a comment - Thanks Kihwal Lee for the reminder. I will merge it to 2.7.1 soon.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Merged to 2.7.1.

          Show
          vinayrpet Vinayakumar B added a comment - Merged to 2.7.1.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7621 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7621/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7621 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7621/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #170 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/170/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #170 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/170/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #904 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/904/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #904 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/904/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2102 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2102/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2102 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2102/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #161 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/161/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #161 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/161/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #171 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/171/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #171 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/171/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2120 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2120/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2120 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2120/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev ed4137cebf27717e9c79eae515b0b83ab6676465) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Vinayakumar B: We deployed this fix to one of our cluster and unfortunately the datanode were still spamming the namenode with the same stack trace as before.
          We debugged the issue and found out that the Datanode were receiving StandbyException wrapped in RemoteException.
          And the patch was checking for StandbyException and not RemoteException.
          I have patch along with testcase with me.
          Let me know if you are fine with me posting the patch.
          Waiting for your reply.

          Show
          shahrs87 Rushabh S Shah added a comment - Vinayakumar B : We deployed this fix to one of our cluster and unfortunately the datanode were still spamming the namenode with the same stack trace as before. We debugged the issue and found out that the Datanode were receiving StandbyException wrapped in RemoteException. And the patch was checking for StandbyException and not RemoteException. I have patch along with testcase with me. Let me know if you are fine with me posting the patch. Waiting for your reply.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Oops, that was a miss.
          Thanks Rushabh S Shah for pointing out.
          Please feel free to post the patch.
          I will revert the earlier commit.

          Show
          vinayrpet Vinayakumar B added a comment - Oops, that was a miss. Thanks Rushabh S Shah for pointing out. Please feel free to post the patch. I will revert the earlier commit.
          Hide
          vinayrpet Vinayakumar B added a comment -

          Assigning to you

          Show
          vinayrpet Vinayakumar B added a comment - Assigning to you
          Hide
          vinayrpet Vinayakumar B added a comment -

          Reverted earlier commit in trunk, branch-2 and branch-2.7.

          Show
          vinayrpet Vinayakumar B added a comment - Reverted earlier commit in trunk, branch-2 and branch-2.7.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7731 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7731/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7731 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7731/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #184 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/184/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #184 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/184/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #918 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/918/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #918 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/918/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2116 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2116/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2116 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2116/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #175 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/175/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #175 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/175/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #185 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/185/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #185 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/185/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2134 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2134/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2134 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2134/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Contributed by Vinayakumar B) (vinayakumarb: rev 318081ccd7af1ec02ec18f35ea95c579326be728) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Hide
          shahrs87 Rushabh S Shah added a comment -

          This patch simply ignores the block when BPServiceActor receives the RemoteException from namenode.
          The namenode got the report once and then it chose not to process that request.
          So it doesn't make much sense to add that request again to queue.

          Show
          shahrs87 Rushabh S Shah added a comment - This patch simply ignores the block when BPServiceActor receives the RemoteException from namenode. The namenode got the report once and then it chose not to process that request. So it doesn't make much sense to add that request again to queue.
          Hide
          kihwal Kihwal Lee added a comment -

          Kicked the precommit build.

          Show
          kihwal Kihwal Lee added a comment - Kicked the precommit build.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 28s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 28s There were no new javac warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 15s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 14s Pre-build of native portion
          -1 hdfs tests 165m 39s Tests failed in hadoop-hdfs.
              208m 17s  



          Reason Tests
          Failed unit tests hadoop.hdfs.tools.TestHdfsConfigFields
            hadoop.tracing.TestTraceAdmin



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731533/HDFS-7916-1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 1dd79ff
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10915/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10915/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10915/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 28s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 15s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 165m 39s Tests failed in hadoop-hdfs.     208m 17s   Reason Tests Failed unit tests hadoop.hdfs.tools.TestHdfsConfigFields   hadoop.tracing.TestTraceAdmin Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731533/HDFS-7916-1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1dd79ff hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10915/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10915/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10915/console This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          The patch didn't cause any test failures.

          • TestHdfsConfigFields is being fixed in HDFS-8371
          • TestTraceAdmin is being fixed in HDFS-8358

          +1 the change looks good.

          Show
          kihwal Kihwal Lee added a comment - The patch didn't cause any test failures. TestHdfsConfigFields is being fixed in HDFS-8371 TestTraceAdmin is being fixed in HDFS-8358 +1 the change looks good.
          Hide
          kihwal Kihwal Lee added a comment -

          Committed to trunk, branch-2 and branch-2.7.

          Show
          kihwal Kihwal Lee added a comment - Committed to trunk, branch-2 and branch-2.7.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7798 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7798/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7798 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7798/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 0s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 41s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 13s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 6s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 3m 14s Pre-build of native portion
          -1 hdfs tests 165m 5s Tests failed in hadoop-hdfs.
              208m 36s  



          Reason Tests
          Failed unit tests hadoop.tracing.TestTraceAdmin
            hadoop.hdfs.tools.TestHdfsConfigFields



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731533/HDFS-7916-1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / b9cebfc
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10916/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10916/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10916/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 13s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 6s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 3m 14s Pre-build of native portion -1 hdfs tests 165m 5s Tests failed in hadoop-hdfs.     208m 36s   Reason Tests Failed unit tests hadoop.tracing.TestTraceAdmin   hadoop.hdfs.tools.TestHdfsConfigFields Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731533/HDFS-7916-1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b9cebfc hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10916/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10916/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10916/console This message was automatically generated.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Thanks Kihwal Lee for reviewing and committing.

          Show
          shahrs87 Rushabh S Shah added a comment - Thanks Kihwal Lee for reviewing and committing.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #925 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/925/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #925 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/925/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #194 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/194/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #194 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/194/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2141 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2141/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2141 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2141/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2123 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2123/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2123 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2123/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #183 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/183/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #183 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/183/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #193 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/193/)
          HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #193 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/193/ ) HDFS-7916 . 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop. Contributed by Rushabh Shah. (kihwal: rev ea11590aad952b5b560a5101d064adf27d8656db) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ErrorReportAction.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReportBadBlockAction.java
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Set affects version to 2.7.0 since this issue is caused by HDFS-7704.

          Show
          ajisakaa Akira Ajisaka added a comment - Set affects version to 2.7.0 since this issue is caused by HDFS-7704 .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          If HDFS-7704 is backported to a branch, this issue should be backported to the same branch as well.

          Show
          ajisakaa Akira Ajisaka added a comment - If HDFS-7704 is backported to a branch, this issue should be backported to the same branch as well.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          I earlier removed HDFS-7916 from the 2.6.1-candidate list given HDFS-7704 was only in 2.7.0.

          Chris Trezzo added it back and so it appeared in my lists. I removed the label again, Chris, please comment on the mailing lists as to why you added it back. If you want it included, please comment there and we can add it after we figure out the why and the dependent tickets.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - I earlier removed HDFS-7916 from the 2.6.1-candidate list given HDFS-7704 was only in 2.7.0. Chris Trezzo added it back and so it appeared in my lists. I removed the label again, Chris, please comment on the mailing lists as to why you added it back. If you want it included, please comment there and we can add it after we figure out the why and the dependent tickets.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Rushabh S Shah and Vinayakumar B,

          Thanks for your earlier work on this.

          I saw that the latest patch does handle ErrorReportAction the same way but there was a comment below:

          Vinayakumar B added a comment - 01/Apr/15 15:03
          Yes. Rushabh S Shah is right. I had already checked about this. ErrorReportAction doesn't need to be handled.

          This is related to Yi Liu's comment (thanks Yi). It seems reasonable to handle both, but because of the above comment, I have a question, did we discover something new such that we made the the change in ErrorReportAction?

          At least this seems to be a discrepancy between the comments and the patch. Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Rushabh S Shah and Vinayakumar B , Thanks for your earlier work on this. I saw that the latest patch does handle ErrorReportAction the same way but there was a comment below: Vinayakumar B added a comment - 01/Apr/15 15:03 Yes. Rushabh S Shah is right. I had already checked about this. ErrorReportAction doesn't need to be handled. This is related to Yi Liu 's comment (thanks Yi). It seems reasonable to handle both, but because of the above comment, I have a question, did we discover something new such that we made the the change in ErrorReportAction? At least this seems to be a discrepancy between the comments and the patch. Thanks.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Yongjun Zhang

          We deployed this fix to one of our cluster and unfortunately the datanode were still spamming the namenode with the same stack trace as before.
          We debugged the issue and found out that the Datanode were receiving StandbyException wrapped in RemoteException.
          And the patch was checking for StandbyException and not RemoteException.

          Inititally we were catching specifically StandbyException. At that time we thought not to catch StandbyException in ErrorReportAction.
          But then we discovered that the namenode was throwing StandbyException wrapped in RemoteException.
          So we chose to ignore all the RemoteException in both the class and just log it as WARN.

          Hope this helps.

          Show
          shahrs87 Rushabh S Shah added a comment - Yongjun Zhang We deployed this fix to one of our cluster and unfortunately the datanode were still spamming the namenode with the same stack trace as before. We debugged the issue and found out that the Datanode were receiving StandbyException wrapped in RemoteException. And the patch was checking for StandbyException and not RemoteException. Inititally we were catching specifically StandbyException. At that time we thought not to catch StandbyException in ErrorReportAction. But then we discovered that the namenode was throwing StandbyException wrapped in RemoteException. So we chose to ignore all the RemoteException in both the class and just log it as WARN. Hope this helps.
          Hide
          yzhangal Yongjun Zhang added a comment -

          HI Rushabh Shah,

          Thanks for your reply and sorry I missed this update until now that I'm looking at a related issue. I created HDFS-9532 when I look at the related code here.

          One thing I'd like to check with you and Vinayakumar B: the HDFS-7916 fix tries to handle StandbyException correctly as reported, but the fix catches RemoteException and did not check whether the exception wrapped by the RemoteException is StandbyException or not. Is it intended to handle all exceptions wrapped by RemoteException the same way as StandbyException? Is there any case that we don't want to do the same? It seems worth some understanding here. Would you guys please comment?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - HI Rushabh Shah , Thanks for your reply and sorry I missed this update until now that I'm looking at a related issue. I created HDFS-9532 when I look at the related code here. One thing I'd like to check with you and Vinayakumar B : the HDFS-7916 fix tries to handle StandbyException correctly as reported, but the fix catches RemoteException and did not check whether the exception wrapped by the RemoteException is StandbyException or not. Is it intended to handle all exceptions wrapped by RemoteException the same way as StandbyException? Is there any case that we don't want to do the same? It seems worth some understanding here. Would you guys please comment? Thanks.
          Hide
          vinayrpet Vinayakumar B added a comment -

          This patch simply ignores the block when BPServiceActor receives the RemoteException from namenode.
          The namenode got the report once and then it chose not to process that request.
          So it doesn't make much sense to add that request again to queue.

          Hi Yongjun Zhang, This is earlier comment from Rushabh Shah about handling entire RemoteException. Do you find any problem here?

          Show
          vinayrpet Vinayakumar B added a comment - This patch simply ignores the block when BPServiceActor receives the RemoteException from namenode. The namenode got the report once and then it chose not to process that request. So it doesn't make much sense to add that request again to queue. Hi Yongjun Zhang , This is earlier comment from Rushabh Shah about handling entire RemoteException. Do you find any problem here?
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Vinayakumar B,

          Thanks for your reply. My question was, this jira suggest to ignore StandbyException, but the fix only check RemoteException. RemoteException could wrap exceptions other than StandbyException. Should we treat the other kind of exceptions wrapped by RemoveException the same as StandbyException (the current fix does that), or we should do more checking to handle StandbyException only this way? Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Vinayakumar B , Thanks for your reply. My question was, this jira suggest to ignore StandbyException, but the fix only check RemoteException. RemoteException could wrap exceptions other than StandbyException. Should we treat the other kind of exceptions wrapped by RemoveException the same as StandbyException (the current fix does that), or we should do more checking to handle StandbyException only this way? Thanks.
          Hide
          kihwal Kihwal Lee added a comment -

          If the namenode in service successfully received a reportBadBlock request and then threw exception (i.e. RemoteException), there is no need to retry, thus no need to unwrap the exception and interpret. But what about SafeModeException? If the active namenode is in safe mode, do we want it to be retried? If yes, we might need to interpret the exception.

          Show
          kihwal Kihwal Lee added a comment - If the namenode in service successfully received a reportBadBlock request and then threw exception (i.e. RemoteException), there is no need to retry, thus no need to unwrap the exception and interpret. But what about SafeModeException? If the active namenode is in safe mode, do we want it to be retried? If yes, we might need to interpret the exception.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks Kihwal Lee, I created HDFS-9540 accordingly. Are there any other exceptions that we need to be aware of? Or maybe we just change the HDFS-7916 fix to interpret RemoteException and handle StandbyException only, and retain the behavior for other exceptions.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks Kihwal Lee , I created HDFS-9540 accordingly. Are there any other exceptions that we need to be aware of? Or maybe we just change the HDFS-7916 fix to interpret RemoteException and handle StandbyException only, and retain the behavior for other exceptions.

            People

            • Assignee:
              shahrs87 Rushabh S Shah
              Reporter:
              vinayrpet Vinayakumar B
            • Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development