Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5616

MR Client-AppMaster RPC max retries on socket timeout is too high.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.2.0
    • Fix Version/s: 3.0.0, 2.3.0
    • Component/s: client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      MAPREDUCE-3811 introduced a separate config key for overriding the max retries applied to RPC connections from the MapReduce Client to the MapReduce Application Master. This was done to make failover from the AM to the MapReduce History Server faster in the event that the AM completes while the client thinks it's still running. However, the RPC client uses a separate setting for socket timeouts, and this one is not overridden. The default for this is 45 retries with a 20-second timeout on each retry. This means that in environments subject to connection timeout instead of connection refused, the client waits 15 minutes for failover.

        Issue Links

          Activity

          Hide
          Chris Nauroth added a comment -

          Linking to MAPREDUCE-3811. I have a patch in progress that will look very similar to that one.

          Show
          Chris Nauroth added a comment - Linking to MAPREDUCE-3811 . I have a patch in progress that will look very similar to that one.
          Hide
          Chris Nauroth added a comment -

          The existing config override is sufficient for "connection refused" errors. It doesn't cover "connection timeout" errors, which is configured separately in the base RPC client code.

          After the AM exits, we would expect connection attempts to cause an immediate connection refused error, not a longer connection timeout error. After all, the packets can get to their destination. There's just no server listening anymore. The reason I saw connection timeouts was a side effect of a feature of Windows Firewall called Stealth Mode. This feature is on by default, and it intentionally drops outbound TCP RST packets for connections initiated against a port with no server listening.

          http://technet.microsoft.com/en-us/library/dd448557%28WS.10%29

          Without getting the RST, the client doesn't know that a connection has been refused, and so it just has to wait for the longer timeout condition. It's possible to disable stealth mode by setting a registry key and restarting the firewall:

          http://msdn.microsoft.com/en-us/library/ff720058.aspx

          That article might be out of date though, because I found that this registry key was really at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\SharedAccess\Parameters\FirewallPolicy\PublicProfile in my environments.

          My only known repro right now is on Windows. I'm leaving this information here for anyone who might notice similar problems on other RPC interactions. I'd still like to get a configuration patch into the client for this.

          Show
          Chris Nauroth added a comment - The existing config override is sufficient for "connection refused" errors. It doesn't cover "connection timeout" errors, which is configured separately in the base RPC client code. After the AM exits, we would expect connection attempts to cause an immediate connection refused error, not a longer connection timeout error. After all, the packets can get to their destination. There's just no server listening anymore. The reason I saw connection timeouts was a side effect of a feature of Windows Firewall called Stealth Mode. This feature is on by default, and it intentionally drops outbound TCP RST packets for connections initiated against a port with no server listening. http://technet.microsoft.com/en-us/library/dd448557%28WS.10%29 Without getting the RST, the client doesn't know that a connection has been refused, and so it just has to wait for the longer timeout condition. It's possible to disable stealth mode by setting a registry key and restarting the firewall: http://msdn.microsoft.com/en-us/library/ff720058.aspx That article might be out of date though, because I found that this registry key was really at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\SharedAccess\Parameters\FirewallPolicy\PublicProfile in my environments. My only known repro right now is on Windows. I'm leaving this information here for anyone who might notice similar problems on other RPC interactions. I'd still like to get a configuration patch into the client for this.
          Hide
          Chris Nauroth added a comment -

          I'm attaching a patch for supporting override of max retries on socket connection timeouts. I chose a default of 3 retries.

          Show
          Chris Nauroth added a comment - I'm attaching a patch for supporting override of max retries on socket connection timeouts. I chose a default of 3 retries.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12612975/MAPREDUCE-5616.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.v2.TestUberAM

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4186//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4186//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12612975/MAPREDUCE-5616.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.TestUberAM +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4186//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4186//console This message is automatically generated.
          Hide
          Chris Nauroth added a comment -

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          Trying to write a test around this timeout condition would likely be very specific to the environment and risk breaking on other environments.

          -1 core tests. The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
          org.apache.hadoop.mapreduce.v2.TestUberAM

          This is the timeout that we've seen elsewhere on TestUberAM. It's unrelated to this patch.

          Show
          Chris Nauroth added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. Trying to write a test around this timeout condition would likely be very specific to the environment and risk breaking on other environments. -1 core tests. The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.TestUberAM This is the timeout that we've seen elsewhere on TestUberAM . It's unrelated to this patch.
          Hide
          Bikas Saha added a comment -

          Looks like a fairly straightforward change for a fairly non-trivial bug. Thanks Chris! +1.

          Show
          Bikas Saha added a comment - Looks like a fairly straightforward change for a fairly non-trivial bug. Thanks Chris! +1.
          Hide
          Chris Nauroth added a comment -

          Thanks for the review, Bikas. I've committed this to trunk and branch-2.

          Show
          Chris Nauroth added a comment - Thanks for the review, Bikas. I've committed this to trunk and branch-2.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #4739 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4739/)
          MAPREDUCE-5616. MR Client-AppMaster RPC max retries on socket timeout is too high. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1542001)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4739 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4739/ ) MAPREDUCE-5616 . MR Client-AppMaster RPC max retries on socket timeout is too high. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1542001 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #392 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/392/)
          MAPREDUCE-5616. MR Client-AppMaster RPC max retries on socket timeout is too high. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1542001)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #392 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/392/ ) MAPREDUCE-5616 . MR Client-AppMaster RPC max retries on socket timeout is too high. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1542001 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1583 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1583/)
          MAPREDUCE-5616. MR Client-AppMaster RPC max retries on socket timeout is too high. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1542001)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1583 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1583/ ) MAPREDUCE-5616 . MR Client-AppMaster RPC max retries on socket timeout is too high. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1542001 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1609 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1609/)
          MAPREDUCE-5616. MR Client-AppMaster RPC max retries on socket timeout is too high. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1542001)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1609 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1609/ ) MAPREDUCE-5616 . MR Client-AppMaster RPC max retries on socket timeout is too high. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1542001 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java

            People

            • Assignee:
              Chris Nauroth
              Reporter:
              Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development