Flume
  1. Flume
  2. FLUME-798

Blocked append interrupted by rotation event

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v0.9.5
    • Fix Version/s: v0.9.5
    • Component/s: Node
    • Labels:
      None

      Description

      Our flume collector seem's to work for a short period of time and then fails with the following exception. When this happens the collector does not reconnect and the system becomes inactive with the processes still running.

      2011-10-14 01:49:47,386 [logicalNode collector0_log_dir-115] ERROR com.cloudera.flume.core.connector.DirectDriver - Closing down due to exception during append calls
      2011-10-14 01:49:47,387 [logicalNode collector0_log_dir-115] INFO com.cloudera.flume.core.connector.DirectDriver - Connector logicalNode collector0_log_dir-115 exited with error: Blocked append interrupted by rotation event
      java.lang.InterruptedException: Blocked append interrupted by rotation event
      at com.cloudera.flume.handlers.rolling.RollSink.append(RollSink.java:209)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.core.MaskDecorator.append(MaskDecorator.java:43)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.debug.InsistentOpenDecorator.append(InsistentOpenDecorator.java:169)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:71)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
      at com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.core.extractors.DateExtractor.append(DateExtractor.java:129)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88)
      at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:133)
      2011-10-14 01:49:47,388 [logicalNode collector0_log_dir-115] INFO com.cloudera.flume.collector.CollectorSource - closed
      2011-10-14 01:49:48,391 [logicalNode collector0_log_dir-115] INFO com.cloudera.flume.handlers.thrift.ThriftEventSource - Closed server on port 36892...
      2011-10-14 01:49:48,391 [logicalNode collector0_log_dir-115] INFO com.cloudera.flume.handlers.thrift.ThriftEventSource - Queue still has 1000 elements ...
      2011-10-14 01:49:58,399 [logicalNode collector0_log_dir-115] WARN com.cloudera.flume.handlers.thrift.ThriftEventSource - Close timed out due to no progress. Closing despite having 1000 values still enqueued
      2011-10-14 01:49:58,399 [logicalNode collector0_log_dir-115] INFO com.cloudera.flume.handlers.rolling.RollSink - closing RollSink 'escapedCustomDfs("hdfs://van-mang-perf-hadoop-namenode1.net:8020/rawLogs/%

      {dateyear}%{datemonth}%{dateday}/%{datehr}00","raw-%{rolltag}" )'
      2011-10-14 01:49:58,400 [logicalNode collector0_log_dir-115] INFO com.cloudera.flume.handlers.rolling.RollSink - double close 'escapedCustomDfs("hdfs://van-mang-perf-hadoop-namenode1.net:8020/rawLogs/%{dateyear}

      -%

      {datemonth}

      -%

      {dateday}

      /%

      {datehr}

      00","raw-%

      {rolltag}

      " )'
      2011-10-14 01:49:58,400 [logicalNode collector0_log_dir-115] ERROR com.cloudera.flume.core.connector.DirectDriver - Exiting driver logicalNode collector0_log_dir-115 in error state CollectorSource | RegexExtractor because Blocked append interrupted by rotation event

      1. 0001-FLUME-798-Modified-RollSink-to-not-cancel-pending-si.patch
        2 kB
        Cameron Gandevia
      2. 0001-FLUME-798-Modified-RollSink-to-not-cancel-pending-si.patch
        2 kB
        Cameron Gandevia
      3. Flume-798.patch
        12 kB
        Prasad Mujumdar
      4. Flume-798.patch.final
        16 kB
        Prasad Mujumdar

        Activity

          People

          • Assignee:
            Prasad Mujumdar
            Reporter:
            Cameron Gandevia
          • Votes:
            7 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development