[ACCUMULO-2716] Duplicate connection loss logging in Writer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.1
• Fix Version/s: 1.5.2, 1.6.0
• Component/s: client
• Labels: logging
• Assignee: Mike Drob
• Reporter: Mike Drob
• Created: 22/Apr/14 17:19
• Updated: 23/Apr/14 17:00
• Resolved: 23/Apr/14 13:18

Description

Running CI with agitation, I see lots of duplicated messages in the monitor whenever a tserver dies.

WARN	Error connecting to tserver1.example.com:10011: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
ERROR	error sending update to tserver1.example.com:10011: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused

These always occur in pairs, at the same millisecond, and coming from the same tserver. I think that they are updates to the metadata table coming from these tservers, like flushes or compactions that fail because the dead server was hosting the corresponding metadata tablet, but it doesn't really matter.

The culprit is in Writer.java where we log-and-rethrow in updateServer():

    } catch (TTransportException e) {
      log.warn("Error connecting to " + server + ": " + e);
      throw e;
    }

and then later log again in update():

      } catch (TException e) {
        log.error("error sending update to " + tabLoc.tablet_location + ": " + e);
        TabletLocator.getLocator(instance, table).invalidateCache(tabLoc.tablet_extent);
      }