[FLUME-375] Unexpected error causes collector to exit "normally"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.9.1
• Fix Version/s: 0.9.5

Environment

Centos 5.5, Dell R610

• Assignee: Unassigned
• Reporter: Disabled imported user
• Created: 06/Dec/10 21:10
• Updated: 05/Nov/14 08:56
• Resolved: 05/Nov/14 08:56

Description

An unexpected error causes the flume collector to exit "normally"

System:
1 master node
2 collector nodes
2 web server nodes

start up collector nodes - successful
start up one web server node - successful
start up second web server node - failure on primary and secondary collector

2010-12-06 11:34:54,190 INFO com.cloudera.flume.agent.LogicalNode: den3db100: Connector stopped: LazyOpenSource | LazyOpenDecorator
2010-12-06 11:34:54,190 INFO com.cloudera.flume.agent.LogicalNode: Connector started: LazyOpenSource | LazyOpenDecorator
2010-12-06 11:34:54,190 INFO com.cloudera.flume.agent.LogicalNode: den3db100_tracking: Connector stopped: LazyOpenSource | LazyOpenDecorator
2010-12-06 11:34:54,239 INFO com.cloudera.flume.handlers.rolling.RollSink: Created RollSink: trigger=[TimeTrigger: maxAge=300000 tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@49f4bcf7] checkPeriodMs = 250 spec='collectorSink'
2010-12-06 11:34:54,241 WARN com.cloudera.flume.handlers.debug.LazyOpenDecorator: Closing a lazy sink that was not logically opened
2010-12-06 11:34:54,241 INFO com.cloudera.flume.agent.LogicalNode: Node config sucessfully set to FlumeConfigData(timestamp:1291643064267, sourceConfig:collectorSource( 35853 ), sinkConfig:collectorSink( "hdfs://den3hdp01/users/flume/collected/tracking", "tracking-", 300000 ), sourceVersion:1291643064267, sinkVersion:1291643064267, flowID:default-flow)
2010-12-06 11:34:54,241 INFO com.cloudera.flume.agent.LogicalNode: Connector started: LazyOpenSource | LazyOpenDecorator
2010-12-06 11:34:54,241 INFO com.cloudera.flume.collector.CollectorSource: opened
2010-12-06 11:34:54,254 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Starting blocking thread pool server on port 35853...
2010-12-06 11:34:54,281 INFO com.cloudera.flume.agent.FlumeNode: Hadoop Security enabled: false
2010-12-06 11:35:18,595 INFO com.cloudera.flume.handlers.debug.StubbornAppendSink: Opening Stubborn Append Sink com.cloudera.flume.handlers.debug.StubbornAppendSink@5ae576d4
2010-12-06 11:35:18,595 INFO com.cloudera.flume.handlers.rolling.RollSink: opening RollSink 'collectorSink'
2010-12-06 11:35:18,746 INFO com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink: Opening hdfs://den3hdp01/users/flume/collected/tracking/tracking-log.00000018.20101206-113518597-0700.435584395296983.seq
2010-12-06 11:35:18,751 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2010-12-06 11:35:18,897 INFO com.cloudera.flume.handlers.hdfs.CustomDfsSink: Opening HDFS file: hdfs://den3hdp01/users/flume/collected/tracking/tracking-log.00000018.20101206-113518597-0700.435584395296983.seq.gz
2010-12-06 11:35:18,897 INFO com.cloudera.flume.handlers.debug.InsistentOpenDecorator: Opened MaskDecorator on try 0
2010-12-06 11:35:18,897 INFO com.cloudera.flume.handlers.debug.StubbornAppendSink: Opening Stubborn Append Sink com.cloudera.flume.handlers.debug.StubbornAppendSink@5ae576d4
2010-12-06 11:35:20,993 ERROR com.cloudera.flume.core.connector.DirectDriver: Driving src/sink failed! LazyOpenSource | LazyOpenDecorator because null
java.lang.NullPointerException
at com.cloudera.flume.handlers.thrift.ThriftEventAdaptor.get(ThriftEventAdaptor.java:131)
at com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:114)
at com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:178)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.handlers.debug.LazyOpenDecorator.append(LazyOpenDecorator.java:69)
at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:92)
2010-12-06 11:35:20,993 INFO com.cloudera.flume.agent.LogicalNode: Connector den3db100_tracking exited with error null
2010-12-06 11:35:20,993 INFO com.cloudera.flume.collector.CollectorSource: closed
2010-12-06 11:35:21,995 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Closed server on port 35853...
2010-12-06 11:35:21,995 INFO com.cloudera.flume.handlers.thrift.ThriftEventSource: Queue still has 182 elements ...
2010-12-06 11:35:32,016 WARN com.cloudera.flume.handlers.thrift.ThriftEventSource: Close timed out due to no progress. Closing despite having 182 values still enqued
2010-12-06 11:35:32,016 INFO com.cloudera.flume.handlers.rolling.RollSink: closing RollSink 'collectorSink'
2010-12-06 11:35:32,016 WARN com.cloudera.flume.handlers.rolling.RollSink: TriggerThread interrupted
2010-12-06 11:35:32,017 INFO com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink: Closing hdfs://den3hdp01/users/flume/collected/tracking/tracking-log.00000018.20101206-113518597-0700.435584395296983.seq
2010-12-06 11:35:32,017 INFO com.cloudera.flume.handlers.hdfs.CustomDfsSink: Closing HDFS file: hdfs://den3hdp01/users/flume/collected/tracking/tracking-log.00000018.20101206-113518597-0700.435584395296983.seq.gz
2010-12-06 11:35:32,023 INFO com.cloudera.flume.handlers.hdfs.CustomDfsSink: done writing compressed file to hdfs
2010-12-06 11:35:32,035 ERROR com.cloudera.flume.agent.LogicalNode: Driver on den3db100_tracking closed LazyOpenSource | LazyOpenDecorator because of null
java.lang.NullPointerException
at com.cloudera.flume.handlers.thrift.ThriftEventAdaptor.get(ThriftEventAdaptor.java:131)
at com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:114)
at com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:178)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.handlers.debug.LazyOpenDecorator.append(LazyOpenDecorator.java:69)
at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:92)

Issue Links

• is related to: FLUME-713 Update Preconditions in EventImpl to provide actionable error messages
• relates to: FLUME-620 Collector fails due to an infinite loop during file-rolling if the body size is greater than 30kb