Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7278

Add a command that allows sysadmins to manually trigger full block reports from a DN

    Details

    • Target Version/s:

      Description

      We should add a command that allows sysadmins to manually trigger full block reports from a DN.

      1. HDFS-7278.002.patch
        22 kB
        Colin P. McCabe
      2. HDFS-7278.003.patch
        25 kB
        Colin P. McCabe
      3. HDFS-7278.004.patch
        25 kB
        Colin P. McCabe
      4. HDFS-7278.005.patch
        25 kB
        Colin P. McCabe

        Activity

        Hide
        sureshms Suresh Srinivas added a comment -

        Colin P. McCabe, can you describe why this is needed so that others have context?

        Show
        sureshms Suresh Srinivas added a comment - Colin P. McCabe , can you describe why this is needed so that others have context?
        Hide
        atm Aaron T. Myers added a comment -

        I think it's a good tool to have in our toolbox to work around possible bugs in NN replica accounting. If an operator suspects such an issue, they might be tempted to restart a DN, or all of the DNs in a cluster, in order to trigger full block reports. It'd be much lighter weight if instead the operator could just manually trigger a full BR instead of having to restart the DN and therefore need to scan all the DN data dirs, etc.

        Show
        atm Aaron T. Myers added a comment - I think it's a good tool to have in our toolbox to work around possible bugs in NN replica accounting. If an operator suspects such an issue, they might be tempted to restart a DN, or all of the DNs in a cluster, in order to trigger full block reports. It'd be much lighter weight if instead the operator could just manually trigger a full BR instead of having to restart the DN and therefore need to scan all the DN data dirs, etc.
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12676481/HDFS-7278.002.patch
        against trunk revision 3b12fd6.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8488//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8488//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676481/HDFS-7278.002.patch against trunk revision 3b12fd6. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8488//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8488//console This message is automatically generated.
        Hide
        sureshms Suresh Srinivas added a comment -

        I think it's a good tool to have in our toolbox to work around possible bugs in NN replica accounting.

        Very interesting. I have not encountered such an issue. If you have details it would be good to share.

        This command must be okay to add.

        Show
        sureshms Suresh Srinivas added a comment - I think it's a good tool to have in our toolbox to work around possible bugs in NN replica accounting. Very interesting. I have not encountered such an issue. If you have details it would be good to share. This command must be okay to add.
        Hide
        atm Aaron T. Myers added a comment - - edited

        Very interesting. I have not encountered such an issue. If you have details it would be good to share.

        I don't really have any firm details, but I do have a suspicion that we may have a bug which results in a block being considered under-replicated (possibly even entirely missing, if all replicas were affected) after a failover, when in fact all of the replicas of the block are just fine on the DNs in the cluster. I will of course share all the details when I figure them out.

        The latest patch looks pretty good to me. Just a few small comments:

        1. Seems like we should restrict this command to require super user privileges. As it stands I believe any user could connect to the DN to trigger a full BR, which though not super harmful doesn't seem right, either.
        2. I think there may be a small race condition in the test case. Since you create a file and then immediately create a spy object to examine calls between the DN and NN, and then assert that no calls of blockReceivedAndDeleted were made, I think it's possible that the DN RPC to send an immediate incremental BR for that file creation might be delayed until after you've created the spy, which would cause the test to unnecessarily fail. I think more reliable would be to create the spy object before creating the file, and then assert that exactly one IBR was sent.
        3. I suspect that the most common use of this command will be to trigger full block reports, not incremental block reports, given that those are sent rather frequently in a busy cluster anyway. Perhaps we should change the default behavior of the command to send a full BR, and change the optional flag to be "-incremental" instead?

        +1 once these are addressed. Thanks, Colin.

        Show
        atm Aaron T. Myers added a comment - - edited Very interesting. I have not encountered such an issue. If you have details it would be good to share. I don't really have any firm details, but I do have a suspicion that we may have a bug which results in a block being considered under-replicated (possibly even entirely missing, if all replicas were affected) after a failover, when in fact all of the replicas of the block are just fine on the DNs in the cluster. I will of course share all the details when I figure them out. The latest patch looks pretty good to me. Just a few small comments: Seems like we should restrict this command to require super user privileges. As it stands I believe any user could connect to the DN to trigger a full BR, which though not super harmful doesn't seem right, either. I think there may be a small race condition in the test case. Since you create a file and then immediately create a spy object to examine calls between the DN and NN, and then assert that no calls of blockReceivedAndDeleted were made, I think it's possible that the DN RPC to send an immediate incremental BR for that file creation might be delayed until after you've created the spy, which would cause the test to unnecessarily fail. I think more reliable would be to create the spy object before creating the file, and then assert that exactly one IBR was sent. I suspect that the most common use of this command will be to trigger full block reports, not incremental block reports, given that those are sent rather frequently in a busy cluster anyway. Perhaps we should change the default behavior of the command to send a full BR, and change the optional flag to be "-incremental" instead? +1 once these are addressed. Thanks, Colin.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Good idea. Would you please add the usage to the document?
        By the way, I suggest to add a command for admin to manually trigger DirectoryScanner as well in order to avoid inconsistency between the disk and the memory.

        Show
        ajisakaa Akira Ajisaka added a comment - Good idea. Would you please add the usage to the document? By the way, I suggest to add a command for admin to manually trigger DirectoryScanner as well in order to avoid inconsistency between the disk and the memory.
        Hide
        cmccabe Colin P. McCabe added a comment -

        ATM wrote: Seems like we should restrict this command to require super user privileges. As it stands I believe any user could connect to the DN to trigger a full BR, which though not super harmful doesn't seem right, either.

        It does check superuser privilieges here:

        +  @Override // ClientDatanodeProtocol
        +  public void triggerBlockReport(BlockReportOptions options)
        +      throws IOException {
        +    checkSuperuserPrivilege();
        

        As non-superuser, you get this:

        woot@keter:/home/cmccabe/hadoop1> /h/bin/hdfs dfsadmin -triggerBlockReport -full 127.0.0.1:6102
        triggerBlockReport error: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied.
                at org.apache.hadoop.hdfs.server.datanode.DataNode.checkSuperuserPrivilege(DataNode.java:757)
                at org.apache.hadoop.hdfs.server.datanode.DataNode.triggerBlockReport(DataNode.java:2915)
        

        I think there may be a small race condition in the test case. Since you create a file and then immediately create a spy object to examine calls between the DN and NN, and then assert that no calls of blockReceivedAndDeleted were made, I think it's possible that the DN RPC to send an immediate incremental BR for that file creation might be delayed until after you've created the spy, which would cause the test to unnecessarily fail. I think more reliable would be to create the spy object before creating the file, and then assert that exactly one IBR was sent.

        Good catch. I will switch it over to creating the spy first and asserting that one IBR is sent.

        I suspect that the most common use of this command will be to trigger full block reports, not incremental block reports, given that those are sent rather frequently in a busy cluster anyway. Perhaps we should change the default behavior of the command to send a full BR, and change the optional flag to be "-incremental" instead?

        OK.

        Akira wrote: By the way, I suggest to add a command for admin to manually trigger DirectoryScanner as well in order to avoid inconsistency between the disk and the memory.

        Sounds reasonable. I think we should do that in a separate JIRA, though.

        Show
        cmccabe Colin P. McCabe added a comment - ATM wrote: Seems like we should restrict this command to require super user privileges. As it stands I believe any user could connect to the DN to trigger a full BR, which though not super harmful doesn't seem right, either. It does check superuser privilieges here: + @Override // ClientDatanodeProtocol + public void triggerBlockReport(BlockReportOptions options) + throws IOException { + checkSuperuserPrivilege(); As non-superuser, you get this: woot@keter:/home/cmccabe/hadoop1> /h/bin/hdfs dfsadmin -triggerBlockReport -full 127.0.0.1:6102 triggerBlockReport error: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied. at org.apache.hadoop.hdfs.server.datanode.DataNode.checkSuperuserPrivilege(DataNode.java:757) at org.apache.hadoop.hdfs.server.datanode.DataNode.triggerBlockReport(DataNode.java:2915) I think there may be a small race condition in the test case. Since you create a file and then immediately create a spy object to examine calls between the DN and NN, and then assert that no calls of blockReceivedAndDeleted were made, I think it's possible that the DN RPC to send an immediate incremental BR for that file creation might be delayed until after you've created the spy, which would cause the test to unnecessarily fail. I think more reliable would be to create the spy object before creating the file, and then assert that exactly one IBR was sent. Good catch. I will switch it over to creating the spy first and asserting that one IBR is sent. I suspect that the most common use of this command will be to trigger full block reports, not incremental block reports, given that those are sent rather frequently in a busy cluster anyway. Perhaps we should change the default behavior of the command to send a full BR, and change the optional flag to be "-incremental" instead? OK. Akira wrote: By the way, I suggest to add a command for admin to manually trigger DirectoryScanner as well in order to avoid inconsistency between the disk and the memory. Sounds reasonable. I think we should do that in a separate JIRA, though.
        Hide
        cmccabe Colin P. McCabe added a comment -

        New patch. I also added an entry in .apt.vm in this version

        Show
        cmccabe Colin P. McCabe added a comment - New patch. I also added an entry in .apt.vm in this version
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12676752/HDFS-7278.003.patch
        against trunk revision 828429d.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8510//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8510//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676752/HDFS-7278.003.patch against trunk revision 828429d. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8510//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8510//console This message is automatically generated.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Thanks for adding the entry.

        @@ -288,7 +289,7 @@ HDFS Commands Guide
                           | of the resource specified by \<key\> on \<host:ipc_port\>.
                           | All other args after are sent to the host.
         *-----------------+-----------------------------------------------------------+
        -| -reconfig <datanode|...> <host:ipc_port> <start|status> | Start
        +| -reconfig \<datanode|...\> \<host:ipc_port\> \<start|status\> | Start
        

        Nice catch to fix the doc build!
        Would you please escape '|' character by '\' to render the table correctly?

        Show
        ajisakaa Akira Ajisaka added a comment - Thanks for adding the entry. @@ -288,7 +289,7 @@ HDFS Commands Guide | of the resource specified by \<key\> on \<host:ipc_port\>. | All other args after are sent to the host. *-----------------+-----------------------------------------------------------+ -| -reconfig <datanode|...> <host:ipc_port> <start|status> | Start +| -reconfig \<datanode|...\> \<host:ipc_port\> \<start|status\> | Start Nice catch to fix the doc build! Would you please escape '|' character by '\' to render the table correctly?
        Hide
        atm Aaron T. Myers added a comment -

        Latest patch looks pretty good to me, though I think I've noticed another race condition in the test. In the case of testing the incremental IBR trigger, I believe that the IBR will be sent promptly regardless of whether or not the heartbeat is set to a long time or the triggerBlockReport command is sent, which means that the test can pass without the admin command actually being run. To test this out, I commented out the call to triggerBlockReport, and indeed that test still passes, although the full BR test correctly fails.

        +1 once this and Akira's doc comment are addressed.

        Show
        atm Aaron T. Myers added a comment - Latest patch looks pretty good to me, though I think I've noticed another race condition in the test. In the case of testing the incremental IBR trigger, I believe that the IBR will be sent promptly regardless of whether or not the heartbeat is set to a long time or the triggerBlockReport command is sent, which means that the test can pass without the admin command actually being run. To test this out, I commented out the call to triggerBlockReport, and indeed that test still passes, although the full BR test correctly fails. +1 once this and Akira's doc comment are addressed.
        Hide
        cmccabe Colin P. McCabe added a comment -

        Add escaping to "reconfig" part of .apt.vm

        Show
        cmccabe Colin P. McCabe added a comment - Add escaping to "reconfig" part of .apt.vm
        Hide
        cmccabe Colin P. McCabe added a comment -

        Aaron T. Myers: Good call. Doing this right is a bit tricky. The latest patch gets it right. I created a synthetic "block deleted" notification which will get sent on the next IBR. But creating this notification doesn't itself trigger an IBR, unlike creating a file.

        Show
        cmccabe Colin P. McCabe added a comment - Aaron T. Myers : Good call. Doing this right is a bit tricky. The latest patch gets it right. I created a synthetic "block deleted" notification which will get sent on the next IBR. But creating this notification doesn't itself trigger an IBR, unlike creating a file.
        Hide
        atm Aaron T. Myers added a comment -

        That looks like it'll do it to me. Thanks, Colin.

        +1 pending Jenkins.

        Show
        atm Aaron T. Myers added a comment - That looks like it'll do it to me. Thanks, Colin. +1 pending Jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12677022/HDFS-7278.005.patch
        against trunk revision 683897f.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:

        org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
        org.apache.hadoop.ha.TestZKFailoverControllerStress

        The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:

        org.apache.hadoop.ha.TestZKFailoverControllerStress

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8534//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8534//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677022/HDFS-7278.005.patch against trunk revision 683897f. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl org.apache.hadoop.ha.TestZKFailoverControllerStress The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: org.apache.hadoop.ha.TestZKFailoverControllerStress +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8534//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8534//console This message is automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12676993/HDFS-7278.004.patch
        against trunk revision a52eb4b.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
        org.apache.hadoop.hdfs.TestDistributedFileSystem

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8529//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8529//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12676993/HDFS-7278.004.patch against trunk revision a52eb4b. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS org.apache.hadoop.hdfs.TestDistributedFileSystem +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8529//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8529//console This message is automatically generated.
        Hide
        cmccabe Colin P. McCabe added a comment -

        It's really odd that the test run for https://issues.apache.org/jira/secure/attachment/12677022/HDFS-7278.005.patch says that the patch "does not include any new or modified tests." You can clearly see that the patch includes a new test, hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java.

        I am re-triggering jenkins.

        Show
        cmccabe Colin P. McCabe added a comment - It's really odd that the test run for https://issues.apache.org/jira/secure/attachment/12677022/HDFS-7278.005.patch says that the patch "does not include any new or modified tests." You can clearly see that the patch includes a new test, hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java. I am re-triggering jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12677022/HDFS-7278.005.patch
        against trunk revision 683897f.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8538//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8538//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677022/HDFS-7278.005.patch against trunk revision 683897f. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8538//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8538//console This message is automatically generated.
        Hide
        cmccabe Colin P. McCabe added a comment -

        Committed to 2.7. Thanks, all.

        Show
        cmccabe Colin P. McCabe added a comment - Committed to 2.7. Thanks, all.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #6353 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6353/)
        HDFS-7278. Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe) (cmccabe: rev baf794dc404ac54f4e8332654eadfac1bebacb8f)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/BlockReportOptions.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientDatanodeProtocol.proto
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6353 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6353/ ) HDFS-7278 . Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe) (cmccabe: rev baf794dc404ac54f4e8332654eadfac1bebacb8f) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/BlockReportOptions.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientDatanodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #726 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/726/)
        HDFS-7278. Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe) (cmccabe: rev baf794dc404ac54f4e8332654eadfac1bebacb8f)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java
        • hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/BlockReportOptions.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientDatanodeProtocol.proto
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #726 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/726/ ) HDFS-7278 . Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe) (cmccabe: rev baf794dc404ac54f4e8332654eadfac1bebacb8f) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/BlockReportOptions.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientDatanodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1940 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1940/)
        HDFS-7278. Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe) (cmccabe: rev baf794dc404ac54f4e8332654eadfac1bebacb8f)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientDatanodeProtocol.proto
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/BlockReportOptions.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1940 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1940/ ) HDFS-7278 . Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe) (cmccabe: rev baf794dc404ac54f4e8332654eadfac1bebacb8f) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientDatanodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/BlockReportOptions.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1915 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1915/)
        HDFS-7278. Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe) (cmccabe: rev baf794dc404ac54f4e8332654eadfac1bebacb8f)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java
        • hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientDatanodeProtocol.proto
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/BlockReportOptions.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1915 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1915/ ) HDFS-7278 . Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe) (cmccabe: rev baf794dc404ac54f4e8332654eadfac1bebacb8f) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientDatanodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/BlockReportOptions.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        dlmarion Dave Marion added a comment -

        Colin P. McCabe Question regarding this new command. If I trigger all of my datanodes to report to the NameNode, will the datanode still honor the dfs.blockreport.initialDelay property to space the reports, or will I flood my NameNode with block reports?

        Show
        dlmarion Dave Marion added a comment - Colin P. McCabe Question regarding this new command. If I trigger all of my datanodes to report to the NameNode, will the datanode still honor the dfs.blockreport.initialDelay property to space the reports, or will I flood my NameNode with block reports?
        Hide
        cmccabe Colin P. McCabe added a comment -

        Hi Dave Marion,

        The -triggerBlockReport command simply triggers a block report for a single datanode. It doesn't honor any delays or settings.

        Show
        cmccabe Colin P. McCabe added a comment - Hi Dave Marion , The -triggerBlockReport command simply triggers a block report for a single datanode. It doesn't honor any delays or settings.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Sangjin Lee backported this to 2.6.1.

        I just pushed the commit to 2.6.1 after running compilation and TestTriggerBlockReport which changed in the patch.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Sangjin Lee backported this to 2.6.1. I just pushed the commit to 2.6.1 after running compilation and TestTriggerBlockReport which changed in the patch.

          People

          • Assignee:
            cmccabe Colin P. McCabe
            Reporter:
            cmccabe Colin P. McCabe
          • Votes:
            0 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development