Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0, 0.23.0
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This jira aims to make client/datanode or datanode/datanode RPC to have a timeout of DataNode#socketTimeout.

      1. hdfsRpcTimeout.patch
        4 kB
        Hairong Kuang
      2. HADOOP-6889-fortrunk-2.patch
        11 kB
        Matt Foley

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          1d 21h 6m 1 Hairong Kuang 05/Aug/10 21:46
          Resolved Resolved Reopened Reopened
          382d 8h 16m 1 Matt Foley 28/Aug/11 01:25
          Reopened Reopened Patch Available Patch Available
          5m 1 Matt Foley 28/Aug/11 01:30
          Patch Available Patch Available Resolved Resolved
          7d 19h 6m 2 Matt Foley 31/Aug/11 01:14
          Resolved Resolved Closed Closed
          103d 5h 5m 1 Konstantin Shvachko 12/Dec/11 06:19
          Allen Wittenauer made changes -
          Fix Version/s 2.0.0-alpha [ 12320353 ]
          Allen Wittenauer made changes -
          Fix Version/s 2.0.0-alpha [ 12320353 ]
          Fix Version/s 0.24.0 [ 12317653 ]
          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #802 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/802/)
          HDFS-1330 and HADOOP-6889. Added additional unit tests. Contributed by John George.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #802 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/802/ ) HDFS-1330 and HADOOP-6889 . Added additional unit tests. Contributed by John George. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #777 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/777/)
          HDFS-1330 and HADOOP-6889. Added additional unit tests. Contributed by John George.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #777 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/777/ ) HDFS-1330 and HADOOP-6889 . Added additional unit tests. Contributed by John George. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Matt Foley made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Matt Foley added a comment -

          +1 for code review. Thanks, John!
          Committed to trunk.

          Also asked Arun if he wanted this in branch-0.23, he said yes.
          Committed to v23.

          Show
          Matt Foley added a comment - +1 for code review. Thanks, John! Committed to trunk. Also asked Arun if he wanted this in branch-0.23, he said yes. Committed to v23.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #821 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/821/)
          HDFS-1330 and HADOOP-6889. Added additional unit tests. Contributed by John George.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #821 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/821/ ) HDFS-1330 and HADOOP-6889 . Added additional unit tests. Contributed by John George. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #888 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/888/)
          HDFS-1330 and HADOOP-6889. Added additional unit tests. Contributed by John George.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #888 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/888/ ) HDFS-1330 and HADOOP-6889 . Added additional unit tests. Contributed by John George. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #811 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/811/)
          HDFS-1330 and HADOOP-6889. Added additional unit tests. Contributed by John George.

          mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #811 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/811/ ) HDFS-1330 and HADOOP-6889 . Added additional unit tests. Contributed by John George. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1163463 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestInterDatanodeProtocol.java
          Hide
          Matt Foley added a comment -

          Thank John, it's his patch I just moved it here so test-patch would run on it correctly; it was refusing to load the patch from within HADOOP-6889. Thanks Uma for pointing out the link to HADOOP-7488. It looks like you've got that under control, so we won't need to hold this (HDFS-1330) open for that.

          Show
          Matt Foley added a comment - Thank John, it's his patch I just moved it here so test-patch would run on it correctly; it was refusing to load the patch from within HADOOP-6889 . Thanks Uma for pointing out the link to HADOOP-7488 . It looks like you've got that under control, so we won't need to hold this ( HDFS-1330 ) open for that.
          Hide
          John George added a comment -

          TestHost2NodesMap failure is unrelated. It seems to have failed on previous builds as well.. Build #1171 is an example.

          Show
          John George added a comment - TestHost2NodesMap failure is unrelated. It seems to have failed on previous builds as well.. Build #1171 is an example.
          Hide
          Uma Maheswara Rao G added a comment -

          Please check the below code snippet from Datanode, where we are not passing the rpcTimeOut.

          DatanodeProtocol dnp = 
                  (DatanodeProtocol)RPC.waitForProxy(DatanodeProtocol.class,
                      DatanodeProtocol.versionID, nnAddr, conf);
          

          This will leads to the issue..HADOOP-7488.

          Show
          Uma Maheswara Rao G added a comment - Please check the below code snippet from Datanode, where we are not passing the rpcTimeOut. DatanodeProtocol dnp = (DatanodeProtocol)RPC.waitForProxy(DatanodeProtocol.class, DatanodeProtocol.versionID, nnAddr, conf); This will leads to the issue.. HADOOP-7488 .
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks Matt for the patch.
          This change is very much required. see the scenario in HADOOP-7488

          Show
          Uma Maheswara Rao G added a comment - Thanks Matt for the patch. This change is very much required. see the scenario in HADOOP-7488
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12491983/HADOOP-6889-fortrunk-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1174//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1174//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1174//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12491983/HADOOP-6889-fortrunk-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1174//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1174//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1174//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12491983/HADOOP-6889-fortrunk-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:

          org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1173//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1173//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1173//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12491983/HADOOP-6889-fortrunk-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1173//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1173//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1173//console This message is automatically generated.
          Matt Foley made changes -
          Attachment HADOOP-6889-fortrunk-2.patch [ 12491983 ]
          Matt Foley made changes -
          Status Reopened [ 4 ] Patch Available [ 10002 ]
          Affects Version/s 0.23.0 [ 12315571 ]
          Fix Version/s 0.23.0 [ 12315571 ]
          Fix Version/s 0.24.0 [ 12317653 ]
          Hide
          Matt Foley added a comment -

          Submitted on behalf of John George. (Moved from HADOOP-6889.)

          Show
          Matt Foley added a comment - Submitted on behalf of John George. (Moved from HADOOP-6889 .)
          Matt Foley made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Assignee Hairong Kuang [ hairong ] John George [ johnvijoe ]
          Hide
          Matt Foley added a comment -

          Re-opening to add enhanced unit testing in v0.23 (moved from HADOOP-6889).

          Show
          Matt Foley added a comment - Re-opening to add enhanced unit testing in v0.23 (moved from HADOOP-6889 ).
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #370 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/370/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #370 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/370/ )
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Hairong Kuang added a comment -

          I've committed this!

          Show
          Hairong Kuang added a comment - I've committed this!
          Hide
          Hairong Kuang added a comment -

          Thanks Sam for reviewing the patch.

          Test results ran on my linux box are posted below:

          ant test-patch:
          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to i
          [exec] nclude 3 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          Ant test did not succeed. Failed tests included TestBlockRecovery, TestHDFSTrash(timeout), and TestBackupNode. But they seemed not related to this patch.

          Show
          Hairong Kuang added a comment - Thanks Sam for reviewing the patch. Test results ran on my linux box are posted below: ant test-patch: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to i [exec] nclude 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Ant test did not succeed. Failed tests included TestBlockRecovery, TestHDFSTrash(timeout), and TestBackupNode. But they seemed not related to this patch.
          Hide
          sam rash added a comment -

          +1 lgtm

          Show
          sam rash added a comment - +1 lgtm
          Hairong Kuang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hairong Kuang made changes -
          Attachment hdfsRpcTimeout.patch [ 12451344 ]
          Hairong Kuang made changes -
          Attachment hdfsRpcTimeout.patch [ 12451282 ]
          Hairong Kuang made changes -
          Attachment hdfsRpcTimeout.patch [ 12451282 ]
          Hide
          Hairong Kuang added a comment -

          A patch for review.

          Show
          Hairong Kuang added a comment - A patch for review.
          Hide
          Hairong Kuang added a comment -

          There are two kinds of communications to a DataNode. Most of them directly uses socket for things like replica reading or writing. A few of them use RPC.

          HDFS-1325 is a problem with file read/write, while this jira is for a problem with RPC communications. We see that a problematic DataNode causes a client stuck in waiting for response forever. Having a timeout should help with this problem.

          Show
          Hairong Kuang added a comment - There are two kinds of communications to a DataNode. Most of them directly uses socket for things like replica reading or writing. A few of them use RPC. HDFS-1325 is a problem with file read/write, while this jira is for a problem with RPC communications. We see that a problematic DataNode causes a client stuck in waiting for response forever. Having a timeout should help with this problem.
          Hide
          Wang Xu added a comment -

          What's the relation between this issue and HDFS-1325 ?

          Show
          Wang Xu added a comment - What's the relation between this issue and HDFS-1325 ?
          Hairong Kuang made changes -
          Field Original Value New Value
          Link This issue is blocked by HADOOP-6889 [ HADOOP-6889 ]
          Hairong Kuang created issue -

            People

            • Assignee:
              John George
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development