mast 14:03:42,946 - configs loaded from ZK.
mast 14:03:43,144 - configs loaded from ZK (not sure why it happended a second time. translation?)
mast 14:03:51,534 - n1 heartbeat - active
mast 14:04:21,584 - n2 heartbeat - active (ignore the previous open / close for n2 - that's FLUME-706)
mast 14:08:44,505 - command to reconfig n2 received
mast 14:08:46,394 - master receives a CancelledKeyException while talking to ZK. Not sure this matters.
mast 14:08:47,526 - configs loaded from ZK (probably due to CancelledKeyException).
node 14:08:47,936 - the collector (logicalNode n2-44) close is requested.
node 14:08:48,940 - n2-44 still has 1000 elements.
mast 14:08:52,896 - n2 heartbeat - closing
node 14:08:58,950 - n2-44 times out while closing due to no progress on close.
node 14:08:59,078 - n2-44 driver completes. n2-44 no longer exists.
node 14:08:59,094 - n2-64 comes into existence - a new instance of the collector.
mast 14:09:02,965 - n2 heartbeat - active (presumably for n2-64)
node - note that there are *no more appearances* of n2 from this point!
mast - n2 continues to heartbeat normally.
node 14:10:48,497 - retransmission of an outstanding ACK group. stale: 60332ms. the first of many.
node 14:10:48,498 - race between SENT vs. SENDING message. Possibly unrelated. Correlated with retrans of ACK above.
node 14:32:18,391 - n1 hits an error appending an event. Root cause: "Blocked append interrupted by rotate."
node 14:32:18,393 - "Input stream pipe closed" from (appears to be related to n1 failure.)
node 14:32:18,397 - n1 error while waiting for exec threads to exit.
node 14:32:18,992 - n1 attempts to ensure WAL thread is closed.
node - n1 - tons of errors about "expected IDLE but timed out in state ACTIVE." followed by...
node - n1 - endless "Attempt X with no progress on wal consumer subsink" (These two alternate.)
node - Roll, heartbeat, checkconfig threads continue.
node 14:45:01,719 - Node shutdown.
mast 14:45:05,585 - master stopped. No exceptions, no node state changes in heartbeats.