Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15856

Cached Connection instances can wind up with addresses never resolved

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0, 1.2.2, 0.98.20, 1.1.6, 2.0.0
    • Component/s: Client
    • Labels:
      None
    • Release Note:
      Hide
      During periods where DNS resolution was not available or not working correctly, we could previously cache unresolved hostnames forever, in some cases preventing further connections to these hosts even when DNS service was restored. With this change, unresolved hostnames will no longer be cached, and will instead throw an UnknownHostException during connection setup.
      Show
      During periods where DNS resolution was not available or not working correctly, we could previously cache unresolved hostnames forever, in some cases preventing further connections to these hosts even when DNS service was restored. With this change, unresolved hostnames will no longer be cached, and will instead throw an UnknownHostException during connection setup.

      Description

      During periods where DNS is not working properly, we can wind up caching connections to master or regionservers where the initial hostname resolution and the resolution is never re-attempted. This means that clients will forever get UnknownHostException for any calls.

      When constructing a BlockingRpcChannelImplementation, we instantiate the InetSocketAddress to use for the connection. This instance is then used in the rpc client connection, where we check isUnresolved() and throw an UnknownHostException if that returns true. However, at this point the rpc channel is already cached in the HConnectionImplementation map of stubs. So at this point it will never be resolved.

      Setting the config for hbase.resolve.hostnames.on.failure masks this issue, since the stub key used is modified to contain the address. However, even in that case, if DNS fails, an rpc channel instance with unresolved ISA will still be cached in the stubs under the hostname only key.

        Attachments

        1. HBASE-15856.001.patch
          8 kB
          Gary Helmling
        2. HBASE-15856.002.patch
          8 kB
          Gary Helmling
        3. HBASE-15856.003.patch
          10 kB
          Gary Helmling
        4. HBASE-15856.addendum.patch
          0.7 kB
          Gary Helmling

          Issue Links

            Activity

              People

              • Assignee:
                ghelmling Gary Helmling
                Reporter:
                ghelmling Gary Helmling
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: