Flume
  1. Flume
  2. FLUME-757

Flume Collector Sink Fails Due to Driver Exception

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: v0.9.4
    • Fix Version/s: v0.9.5
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      I am experiencing a weird issue with my Flume configuration. Occasionally when I start my agents they start delivering messages to the collectors and after awhile I receive the following exceptions and stop receiving messages.

      The collectors are configured as follows.

      Source: collectorSource(36892)
      Sink: {regex("^(?:\\n)?(\\d\\d\\d\\d-\\d\\d-\\d\\d\\s\\d\\d:\\d
      d)", 1, "date") => {exDate("date", "yyyy-MM-dd HH:mm") => collectorSink("hdfs://hadoop-namenode1:8020/logs/%

      {dateyear}

      -%

      {datemonth}

      -%

      {dateday}

      /%

      {datehr}

      00/%

      {host}

      ", "log-")}}

      The agents are configured as follows.

      Source: tailDir("/data1/logs", delim="\\n\\d\\d\\d
      d", delimMode="next")
      Sink: agentDFOChain("flume-collector1:36892", "flume-collector2:36892")

      The following exceptions are from my collectors.

      INFO com.cloudera.flume.handlers.debug.InsistentAppendDecorator - Failed due to unexpected runtime exception during append attempt
      java.lang.NullPointerException
      at com.cloudera.flume.handlers.hdfs.CustomDfsSink.close(CustomDfsSink.java:88)
      at com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.close(EscapedCustomDfsSink.java:132)
      at com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56)
      at com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
      at com.cloudera.flume.collector.CollectorSink$RollDetectDeco.close(CollectorSink.java:171)
      at com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:331)
      at com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
      at com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
      at com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175)
      at com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
      at com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:96)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
      at com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.core.extractors.DateExtractor.append(DateExtractor.java:129)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88)
      at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
      ERROR com.cloudera.flume.core.connector.DirectDriver - Closing down due to exception during append calls
      java.lang.NullPointerException
      at com.cloudera.flume.handlers.hdfs.CustomDfsSink.close(CustomDfsSink.java:88)
      at com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.close(EscapedCustomDfsSink.java:132)
      at com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56)
      at com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
      at com.cloudera.flume.collector.CollectorSink$RollDetectDeco.close(CollectorSink.java:171)
      at com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:331)
      at com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
      at com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
      at com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175)
      at com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
      at com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:96)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
      at com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.core.extractors.DateExtractor.append(DateExtractor.java:129)
      at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
      at com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88)
      at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)

      The following are from my agents

      ERROR com.cloudera.flume.core.connector.DirectDriver - Expected IDLE but timed out in state ACTIVE
      INFO com.cloudera.flume.agent.diskfailover.DiskFailoverDeco - Closing disk failover log, subsink still making progre
      ERROR com.cloudera.flume.agent.LogicalNode - Forcing driver to exit uncleanly
      INFO com.cloudera.flume.agent.LogicalNode - Node config successfully set to com.cloudera.flume.conf.FlumeConfigData@19a0feb
      WARN com.cloudera.flume.handlers.debug.LazyOpenDecorator - Closing a lazy sink that was not logically opened
      INFO com.cloudera.flume.handlers.text.TailDirSource - added file /data1/mutator/logs/one.log
      INFO com.cloudera.flume.handlers.rolling.RollSink - Created RollSink: trigger=[TimeTrigger: maxAge=10000 tagger=com.
      cloudera.flume.handlers.rolling.ProcessTagger@6279d] checkPeriodMs = 250 spec='NaiveFileFailover'
      INFO com.cloudera.flume.handlers.rolling.RollSink - opening RollSink 'NaiveFileFailover'
      ERROR com.cloudera.flume.agent.diskfailover.DiskFailoverDeco - WAL drain thread interrupted
      java.lang.InterruptedException
      at java.lang.Object.wait(Native Method)
      at com.cloudera.flume.core.connector.DirectDriver.waitForAtLeastState(DirectDriver.java:308)
      at com.cloudera.flume.agent.diskfailover.DiskFailoverDeco.ensureClosedDrainDriver(DiskFailoverDeco.java:129)
      at com.cloudera.flume.agent.diskfailover.DiskFailoverDeco.close(DiskFailoverDeco.java:177)
      at com.cloudera.flume.core.BackOffFailOverSink.close(BackOffFailOverSink.java:165)
      at com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56)
      at com.cloudera.flume.agent.AgentFailChainSink.close(AgentFailChainSink.java:98)
      at com.cloudera.flume.core.CompositeSink.close(CompositeSink.java:56)
      at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:126)

        Activity

        Cameron Gandevia created issue -
        Ashish Paliwal made changes -
        Field Original Value New Value
        Resolution Won't Fix [ 2 ]
        Fix Version/s v0.9.5 [ 12317558 ]
        Status Open [ 1 ] Resolved [ 5 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Cameron Gandevia
          • Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development