[HBASE-4283] HBaseAdmin never recovers from restarted cluster

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.92.0
• Fix Version/s: 0.92.0
• Component/s: Client
• Hadoop Flags: Reviewed
• Assignee: Lars Hofhansl
• Reporter: Lars Hofhansl
• Created: 29/Aug/11 22:29
• Updated: 20/Nov/15 11:52
• Resolved: 31/Aug/11 21:09

Description

While testing common scenarios that we might encounter I found that HBaseAdmin does not recover from a restarted cluster.

It turns out HBaseClient.Connection.stop() is send into an endless loop here:

    // wait until all connections are closed
    while (!connections.isEmpty()) {
      try {
        Thread.sleep(100);
      } catch (InterruptedException ignored) {
      }
    }

The reason is that PoolMap.remove(k,v) does not remove empty pools, and hence connections.isEmpty() is never true if there ever was any connection in there.
My fix is to remove the pool from the poolMap when it is empty. (Alternatively one could change PoolMap.isEmpty() to also look inside of all pools and see if their size is 0).

When I fixed that I noticed that if the master wasn't running when HBaseAdmin is created it also will not recover from that.
Even creating a new HBaseAdmin from the same Configuration will still use the old stale HConnection.

In that case a MasterNotRunningException is thrown, which is not handled in HBaseAdmin's constructor.

The HConnection handling in HConnectionManager is funky. There should never be a closed connection in the HBASE_INSTANCES.
I might look at that as well but in a separate issue.

Issue Links

• relates to: HBASE-5058 Allow HBaseAdmin to use an existing connection