[FLUME-543] TailThread exits on repeated WAL failures

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.9.3
• Fix Version/s: 0.9.5
• Component/s: Sinks+Sources

Environment

Amazon EC2 Ubuntu 10.04 LTS

• Assignee: Jonathan Hsieh
• Reporter: Disabled imported user
• Created: 01/Mar/11 01:17
• Updated: 25/Aug/11 11:40
• Resolved: 25/Aug/11 11:40

Description

On repeated failures:

2011-02-28 23:37:26,908 ERROR com.cloudera.flume.core.connector.DirectDriver: Driving src/sink failed! LazyOpenSource | LazyOpenDecorator because java.lang.IllegalStateException: Expected state to be LOGGED but was SENT
java.io.IOException: java.lang.IllegalStateException: Expected state to be LOGGED but was SENT
	at com.cloudera.flume.agent.durability.NaiveFileWALDeco$1.fireError(NaiveFileWALDeco.java:254)
	at com.cloudera.flume.core.connector.DirectDriver.fireError(DirectDriver.java:216)
	at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:102)
Caused by: java.lang.IllegalStateException: Expected state to be LOGGED but was SENT
	at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
	at com.cloudera.flume.agent.durability.NaiveFileWALManager.changeState(NaiveFileWALManager.java:410)
	at com.cloudera.flume.agent.durability.NaiveFileWALManager.getUnackedSource(NaiveFileWALManager.java:557)
	at com.cloudera.flume.agent.durability.WALSource.next(WALSource.java:104)
	at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:89)

The tailer exits and the master reports all nodes in ERROR state.

2011-02-28 23:37:27,917 INFO com.cloudera.flume.handlers.text.TailSource: TailThread has exited

Full log file attached.

Rob

Issue Links

• is blocked by: FLUME-746 Correct the behavior and logging messages about states transition of wal chunks on retry
• is related to: FLUME-637 Flume Node deadlocks on getting reports/metrics info.
• is related to: FLUME-657 Improve debugging output of FlumeConfigData
• relates to: FLUME-745 Fix Race condition in NaiveFileWALDeco and retransmit logic