[FLUME-2209] AsyncHBaseSink will never recover if the column family does not exists for the first start

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.5.0
• Component/s: Sinks+Sources
• Assignee: Ashish Paliwal
• Reporter: nijel
• Created: 09/Oct/13 10:42
• Updated: 04/May/14 04:18
• Resolved: 10/Dec/13 04:06

Description

Hi,
I am facing one issue. Initial analysis is as follows

Using AsyncHBaseSink. During startup the configured column family does not exists and it is throwing exception as

09 Oct 2013 14:45:56,691 ERROR [lifecycleSupervisor-1-2] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)  - Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@5470be88 counterGroup:{ name:null counters:{} } } - Exception follows.
org.apache.flume.FlumeException: Could not start sink. Table or column family does not exist in Hbase.
	at org.apache.flume.sink.hbase.AsyncHBaseSink.start(AsyncHBaseSink.java:384)
	at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
	at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
	at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

After this, the sink is died and LifecycleSupervisor is trying to start again

Want to transition SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2d7aece8 counterGroup:{ name:null counters:{} } } from IDLE to START (failures:23)

Here it is getting exception as

09 Oct 2013 14:49:26,043 ERROR [lifecycleSupervisor-1-2] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)  - Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2d7aece8 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.IllegalArgumentException: Please call stop before calling start on an old instance.
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
	at org.apache.flume.sink.hbase.AsyncHBaseSink.start(AsyncHBaseSink.java:344)
	at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
	at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
	at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

The issue i think is we are not closing the client when the exception comes.

 if(fail.get()){
      sinkCounter.incrementConnectionFailedCount();
      throw new FlumeException(
          "Could not start sink. " +
          "Table or column family does not exist in Hbase.");
    } else {
      open = true;
    }

is this a issue ? or am i missing something ?

Issue Links

• links to: Review Request