Flume
  1. Flume
  2. FLUME-2209

AsyncHBaseSink will never recover if the column family does not exists for the first start

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: v1.4.0
    • Fix Version/s: v1.5.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Hi,
      I am facing one issue. Initial analysis is as follows

      Using AsyncHBaseSink. During startup the configured column family does not exists and it is throwing exception as

      09 Oct 2013 14:45:56,691 ERROR [lifecycleSupervisor-1-2] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)  - Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@5470be88 counterGroup:{ name:null counters:{} } } - Exception follows.
      org.apache.flume.FlumeException: Could not start sink. Table or column family does not exist in Hbase.
      	at org.apache.flume.sink.hbase.AsyncHBaseSink.start(AsyncHBaseSink.java:384)
      	at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
      	at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
      	at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      

      After this, the sink is died and LifecycleSupervisor is trying to start again

      Want to transition SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2d7aece8 counterGroup:{ name:null counters:{} } } from IDLE to START (failures:23)
      

      Here it is getting exception as

      09 Oct 2013 14:49:26,043 ERROR [lifecycleSupervisor-1-2] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)  - Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2d7aece8 counterGroup:{ name:null counters:{} } } - Exception follows.
      java.lang.IllegalArgumentException: Please call stop before calling start on an old instance.
      	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
      	at org.apache.flume.sink.hbase.AsyncHBaseSink.start(AsyncHBaseSink.java:344)
      	at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
      	at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
      	at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      

      The issue i think is we are not closing the client when the exception comes.

       if(fail.get()){
            sinkCounter.incrementConnectionFailedCount();
            throw new FlumeException(
                "Could not start sink. " +
                "Table or column family does not exist in Hbase.");
          } else {
            open = true;
          }
      

      is this a issue ? or am i missing something ?

      1. FLUME-2209-0.patch
        0.8 kB
        Ashish Paliwal

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Ashish Paliwal
              Reporter:
              nijel
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development