Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0, 0.23.0
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This jira aims to make client/datanode or datanode/datanode RPC to have a timeout of DataNode#socketTimeout.

      1. HADOOP-6889-fortrunk-2.patch
        11 kB
        Matt Foley
      2. hdfsRpcTimeout.patch
        4 kB
        Hairong Kuang

        Issue Links

          Activity

          Hide
          Wang Xu added a comment -

          What's the relation between this issue and HDFS-1325 ?

          Show
          Wang Xu added a comment - What's the relation between this issue and HDFS-1325 ?
          Hide
          Hairong Kuang added a comment -

          There are two kinds of communications to a DataNode. Most of them directly uses socket for things like replica reading or writing. A few of them use RPC.

          HDFS-1325 is a problem with file read/write, while this jira is for a problem with RPC communications. We see that a problematic DataNode causes a client stuck in waiting for response forever. Having a timeout should help with this problem.

          Show
          Hairong Kuang added a comment - There are two kinds of communications to a DataNode. Most of them directly uses socket for things like replica reading or writing. A few of them use RPC. HDFS-1325 is a problem with file read/write, while this jira is for a problem with RPC communications. We see that a problematic DataNode causes a client stuck in waiting for response forever. Having a timeout should help with this problem.
          Hide
          Hairong Kuang added a comment -

          A patch for review.

          Show
          Hairong Kuang added a comment - A patch for review.
          Hide
          sam rash added a comment -

          +1 lgtm

          Show
          sam rash added a comment - +1 lgtm
          Hide
          Hairong Kuang added a comment -

          Thanks Sam for reviewing the patch.

          Test results ran on my linux box are posted below:

          ant test-patch:
          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to i
          [exec] nclude 3 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          Ant test did not succeed. Failed tests included TestBlockRecovery, TestHDFSTrash(timeout), and TestBackupNode. But they seemed not related to this patch.

          Show
          Hairong Kuang added a comment - Thanks Sam for reviewing the patch. Test results ran on my linux box are posted below: ant test-patch: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to i [exec] nclude 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Ant test did not succeed. Failed tests included TestBlockRecovery, TestHDFSTrash(timeout), and TestBackupNode. But they seemed not related to this patch.
          Hide
          Hairong Kuang added a comment -

          I've committed this!

          Show
          Hairong Kuang added a comment - I've committed this!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #370 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/370/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #370 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/370/ )
          Hide
          Matt Foley added a comment -

          Re-opening to add enhanced unit testing in v0.23 (moved from HADOOP-6889).

          Show
          Matt Foley added a comment - Re-opening to add enhanced unit testing in v0.23 (moved from HADOOP-6889 ).
          Hide
          Matt Foley added a comment -

          Submitted on behalf of John George. (Moved from HADOOP-6889.)

          Show
          Matt Foley added a comment - Submitted on behalf of John George. (Moved from HADOOP-6889 .)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12491983/HADOOP-6889-fortrunk-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:

          org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1173//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1173//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1173//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12491983/HADOOP-6889-fortrunk-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1173//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1173//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1173//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12491983/HADOOP-6889-fortrunk-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1174//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1174//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1174//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12491983/HADOOP-6889-fortrunk-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1174//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1174//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1174//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks Matt for the patch.
          This change is very much required. see the scenario in HADOOP-7488

          Show
          Uma Maheswara Rao G added a comment - Thanks Matt for the patch. This change is very much required. see the scenario in HADOOP-7488
          Hide
          Uma Maheswara Rao G added a comment -

          Please check the below code snippet from Datanode, where we are not passing the rpcTimeOut.

          DatanodeProtocol dnp = 
                  (DatanodeProtocol)RPC.waitForProxy(DatanodeProtocol.class,
                      DatanodeProtocol.versionID, nnAddr, conf);
          

          This will leads to the issue..HADOOP-7488.

          Show
          Uma Maheswara Rao G added a comment - Please check the below code snippet from Datanode, where we are not passing the rpcTimeOut. DatanodeProtocol dnp = (DatanodeProtocol)RPC.waitForProxy(DatanodeProtocol.class, DatanodeProtocol.versionID, nnAddr, conf); This will leads to the issue.. HADOOP-7488 .
          Hide
          John George added a comment -

          TestHost2NodesMap failure is unrelated. It seems to have failed on previous builds as well.. Build #1171 is an example.

          Show
          John George added a comment - TestHost2NodesMap failure is unrelated. It seems to have failed on previous builds as well.. Build #1171 is an example.
          Hide
          Matt Foley added a comment -

          Thank John, it's his patch I just moved it here so test-patch would run on it correctly; it was refusing to load the patch from within HADOOP-6889. Thanks Uma for pointing out the link to HADOOP-7488. It looks like you've got that under control, so we won't need to hold this (HDFS-1330) open for that.

          Show
          Matt Foley added a comment - Thank John, it's his patch I just moved it here so test-patch would run on it correctly; it was refusing to load the patch from within HADOOP-6889 . Thanks Uma for pointing out the link to HADOOP-7488 . It looks like you've got that under control, so we won't need to hold this ( HDFS-1330 ) open for that.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #811 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/811/)
          HDFS-1330 and HADOOP-6889. Added additional unit tests. Contributed by John George.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #811 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/811/ ) HDFS-1330 and HADOOP-6889 . Added additional unit tests. Contributed by John George. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #888 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/888/)
          HDFS-1330 and HADOOP-6889. Added additional unit tests. Contributed by John George.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #888 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/888/ ) HDFS-1330 and HADOOP-6889 . Added additional unit tests. Contributed by John George. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #821 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/821/)
          HDFS-1330 and HADOOP-6889. Added additional unit tests. Contributed by John George.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #821 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/821/ ) HDFS-1330 and HADOOP-6889 . Added additional unit tests. Contributed by John George. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Hide
          Matt Foley added a comment -

          +1 for code review. Thanks, John!
          Committed to trunk.

          Also asked Arun if he wanted this in branch-0.23, he said yes.
          Committed to v23.

          Show
          Matt Foley added a comment - +1 for code review. Thanks, John! Committed to trunk. Also asked Arun if he wanted this in branch-0.23, he said yes. Committed to v23.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #777 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/777/)
          HDFS-1330 and HADOOP-6889. Added additional unit tests. Contributed by John George.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #777 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/777/ ) HDFS-1330 and HADOOP-6889 . Added additional unit tests. Contributed by John George. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #802 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/802/)
          HDFS-1330 and HADOOP-6889. Added additional unit tests. Contributed by John George.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #802 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/802/ ) HDFS-1330 and HADOOP-6889 . Added additional unit tests. Contributed by John George. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java

            People

            • Assignee:
              John George
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development