Hadoop Common
  1. Hadoop Common
  2. HADOOP-9503

Remove sleep between IPC client connect timeouts

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.0-beta
    • Fix Version/s: 2.1.0-beta
    • Component/s: ipc
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-9106 introduced a configurable retry timeout when there are socket timeouts errors.

      Since the IPC client is used by a wide variety of applications including realtime/online ones (like hbase + hdfs), we sometimes need fast fail timeouts which are < 3 seconds. Achieving such a timeout for connect is very difficult with a 1 second sleep in b/w retries. It would be good to remove this sleep.

      1. 9503.patch
        0.7 kB
        Varun Sharma

        Activity

        Hide
        Tsz Wo Nicholas Sze added a comment -

        Hi Varun, the IPC retry policy is already configurable; see HDFS-3504.

        Show
        Tsz Wo Nicholas Sze added a comment - Hi Varun, the IPC retry policy is already configurable; see HDFS-3504 .
        Hide
        Varun Sharma added a comment -

        Tsz Wo Nicholas Sze
        Hi Nicholas,

        This configures things when you have IOException but for SocketTimeout(s) we had a hard coded 1s sleep with a hard coded 20second connect timeout and hard coded 45 retries. Both were fixed but still there is a constant sleep of 1 second b/w retries for socket timeout errors (which you can get when a host black holed - like firewalled to simulate network disasters).

        Show
        Varun Sharma added a comment - Tsz Wo Nicholas Sze Hi Nicholas, This configures things when you have IOException but for SocketTimeout(s) we had a hard coded 1s sleep with a hard coded 20second connect timeout and hard coded 45 retries. Both were fixed but still there is a constant sleep of 1 second b/w retries for socket timeout errors (which you can get when a host black holed - like firewalled to simulate network disasters).
        Hide
        Varun Sharma added a comment -

        The other option is to simply remove the sleep - I can understand a need for having a sleep b/w true connection failures (like IOErrors) but for timeouts we may just want to retry immediately since we already waited for a loong time for a response.

        Show
        Varun Sharma added a comment - The other option is to simply remove the sleep - I can understand a need for having a sleep b/w true connection failures (like IOErrors) but for timeouts we may just want to retry immediately since we already waited for a loong time for a response.
        Hide
        Daryn Sharp added a comment -

        I agree there's probably not much value in sleeping further after a timeout.

        Show
        Daryn Sharp added a comment - I agree there's probably not much value in sleeping further after a timeout.
        Hide
        Varun Sharma added a comment -

        Trivial patch removing the sleep...

        Show
        Varun Sharma added a comment - Trivial patch removing the sleep...
        Hide
        Daryn Sharp added a comment -

        +1 Although I defer to Nicholas also approving.

        Show
        Daryn Sharp added a comment - +1 Although I defer to Nicholas also approving.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12580566/9503.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2476//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2476//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580566/9503.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2476//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2476//console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 removing the sleep for connection timeout sounds good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 removing the sleep for connection timeout sounds good.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Varun!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Varun!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3666 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3666/)
        HADOOP-9503. Remove sleep between IPC client connect timeouts. Contributed by Varun Sharma (Revision 1475959)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1475959
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3666 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3666/ ) HADOOP-9503 . Remove sleep between IPC client connect timeouts. Contributed by Varun Sharma (Revision 1475959) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1475959 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Also, thanks Daryn for reviewing this.

        Show
        Tsz Wo Nicholas Sze added a comment - Also, thanks Daryn for reviewing this.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #195 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/195/)
        HADOOP-9503. Remove sleep between IPC client connect timeouts. Contributed by Varun Sharma (Revision 1475959)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1475959
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #195 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/195/ ) HADOOP-9503 . Remove sleep between IPC client connect timeouts. Contributed by Varun Sharma (Revision 1475959) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1475959 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1384 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1384/)
        HADOOP-9503. Remove sleep between IPC client connect timeouts. Contributed by Varun Sharma (Revision 1475959)

        Result = FAILURE
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1475959
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1384 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1384/ ) HADOOP-9503 . Remove sleep between IPC client connect timeouts. Contributed by Varun Sharma (Revision 1475959) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1475959 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1411 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1411/)
        HADOOP-9503. Remove sleep between IPC client connect timeouts. Contributed by Varun Sharma (Revision 1475959)

        Result = SUCCESS
        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1475959
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1411 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1411/ ) HADOOP-9503 . Remove sleep between IPC client connect timeouts. Contributed by Varun Sharma (Revision 1475959) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1475959 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java

          People

          • Assignee:
            Varun Sharma
            Reporter:
            Varun Sharma
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development