Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2961

Make TaildirSource work with multiline

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 1.7.0
    • None
    • Sinks+Sources
    • None

    Description

      TaidirSource defaults to LINE, this has issue when multiline log events like stack traces and have request/responses. Following part is Java traceback logs. We expect to have log line start regex Key to aggregate all the log lines till the next regex key is found.
      2016-07-16 14:59:43,956 ERROR lifecycleSupervisor-1-7 LifecycleSupervisor.run - Unable to start EventDrivenSourceRunner: { source:cn.yottabyte.flume.source.http.HTTPSource

      {name:sourceHttp,state:IDLE}

      } - Exception follows.
      java.lang.IllegalStateException: Running HTTP Server found in source: sourceHttp before I started one. Will not attempt to start.
      at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
      at cn.yottabyte.flume.source.http.HTTPSource.startHttpSourceServer(HTTPSource.java:170)
      at cn.yottabyte.flume.source.http.HTTPSource.start(HTTPSource.java:166)
      at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
      at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

      Attachments

        1. FLUME-2961_5.patch
          40 kB
          tinawenqiao
        2. FLUME-2961_4.patch
          35 kB
          tinawenqiao
        3. FLUME-2961_2.patch
          34 kB
          tinawenqiao
        4. FLUME-2961_1.patch
          17 kB
          tinawenqiao

        Issue Links

          Activity

            People

              Unassigned Unassigned
              wenqiao tinawenqiao
              Votes:
              3 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated: