[ACCUMULO-2480] ha fail-failover failure

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.0
• Component/s: master, tserver

Environment

running continuous ingest on a 74-node HA NN hadoop 2.3 cluster, 1.6.0-SNAPSHOT.

• Assignee: Eric C. Newton
• Reporter: Eric C. Newton
• Created: 14/Mar/14 20:07
• Updated: 14/Aug/15 20:37
• Resolved: 29/Sep/14 13:37

Description

Ran service network stop on the active NN. The service failed to switch over since the fencing script on the standby failed to run (sshfence).

After the network interface was re-established, the standby took over.

However, accumulo ingest began to have very long hold times since the standby was not providing service for several minutes.

The master attempted to shutdown the tablet servers with hold time.

The filesystem hook closed the filesystem, and the servers got stuck endlessly trying to write to the WAL.

Even after the NN was active, because the filesytem was closed, attempts to get a new WAL continued to fail.

• why didn't the tablet servers stop?
• WAL loop should be able to terminate if they see an IOException that indicates that the filesystem is closed

Issue Links

• breaks: ACCUMULO-3937 Hard-coded HDFS failure tolerance
• breaks: ACCUMULO-3864 MiniAccumuloClusterImpl.getMiniDfs should be synchronized
• relates to: ACCUMULO-3963 Incremental backoff on inability to write to HDFS