Hadoop Common
  1. Hadoop Common
  2. HADOOP-7466 Hadoop Log improvement Umbrella
  3. HADOOP-7469

add a standard handler for socket connection problems which improves diagnostics

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.203.0, 0.23.0
    • Fix Version/s: 0.23.0, 0.24.0
    • Component/s: util
    • Labels:

      Description

      connection refused, connection timed out, no route to host, etc, are classic IOExceptions that can be raised in a lot of parts of the code. The standard JDK exceptions are useless for debugging as they

      1. don't include the destination (host, port) that can be used in diagnosing service dead/blocked problems
      2. don't include any source hostname that can be used to handle routing issues
      3. assume the reader understands the TCP stack.
        It's obvious from the -user lists that a lot of people hit these problems and don't know how to fix them. Sometimes the source has been patched to insert the diagnostics, but it may be convenient to have a single method to translate some
        SocketException processIOException(SocketException e, String destHost, int destPort) {
          String localhost = getLocalHostname();
          String details = "From "+ localhost +" to "+ desthost + ":"+destPort;
          if (e instanceof ConnectException) {
            return new ConnectException(details 
                    + " -- see http://wiki.apache.org/hadoop/ConnectionRefused --" + e, e);
          }
          if (e instanceof UnknownHostException) {
            return new UnknownHostException(details 
                    + " -- see http://wiki.apache.org/hadoop/UnknownHost --" + e, e);
          }
          // + handlers for other common socket exceptions
          
        //and a default that returns an unknown class unchanged
          return e;
        }
          
        

      Testing: try to connect to an unknown host, a local port that isn't live, etc. It's hard to replicate all failures consistently. It may be simpler just to verify that if you pass in a specific exception, the string is expanded and the class is unchanged.

      This code could then be patched in to places where IO takes place. Note that Http Components and HttpClient libs already add some destination details on some operation failures, with their own HttpException tree: it's simplest to leave these alone.

      1. HADOOP-7469-b023.patch
        17 kB
        Steve Loughran
      2. HADOOP-7469.patch
        18 kB
        Steve Loughran
      3. HADOOP-7469.patch
        17 kB
        Steve Loughran
      4. HADOOP-7469-3.patch
        11 kB
        Steve Loughran
      5. HADOOP-7466-connection-handler.patch
        11 kB
        Steve Loughran
      6. HADOOP-7469.patch
        10 kB
        Uma Maheswara Rao G

        Activity

          People

          • Assignee:
            Steve Loughran
            Reporter:
            Steve Loughran
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 2h
              2h
              Remaining:
              Remaining Estimate - 2h
              2h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development