[FLUME-2961] Make TaildirSource work with multiline

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.7.0
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: tinawenqiao
• Created: 26/Jul/16 08:21
• Updated: 01/Jun/22 06:43

Description

TaidirSource defaults to LINE, this has issue when multiline log events like stack traces and have request/responses. Following part is Java traceback logs. We expect to have log line start regex Key to aggregate all the log lines till the next regex key is found.
2016-07-16 14:59:43,956 ERROR lifecycleSupervisor-1-7 LifecycleSupervisor.run - Unable to start EventDrivenSourceRunner: { source:cn.yottabyte.flume.source.http.HTTPSource

{name:sourceHttp,state:IDLE}

} - Exception follows.
java.lang.IllegalStateException: Running HTTP Server found in source: sourceHttp before I started one. Will not attempt to start.
at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at cn.yottabyte.flume.source.http.HTTPSource.startHttpSourceServer(HTTPSource.java:170)
at cn.yottabyte.flume.source.http.HTTPSource.start(HTTPSource.java:166)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Issue Links

• is duplicated by: FLUME-2800 Multiline log events for Taildir Source
• links to: Review Board