[FLUME-3423] Taildir source will throw unnecessay exception when restart agent

• Type: Bug
• Status: Open
• Priority: Trivial
• Resolution: Unresolved
• Affects Version/s: 1.9.0
• Component/s: Sinks+Sources
• Labels: patch
• Flags: Patch

Language

Java

• Assignee: Unassigned
• Reporter: Jasper D
• Created: 15/Apr/22 16:51
• Updated: 01/Jun/22 19:01

Description

Phenomenon

When we use Taildir source, it will auto create an empty tail position file at start time. But if the filegroup directory is empty,  it has no file to tail and then we restart agent, it will throw an EOFException. Although it has little effect on functionality, it will annoy users.

Recur

This problem can esaily recur.

We can use the example.conf from flume user guide directly.

//代码占位符
a1.sources = r1
a1.channels = c1
a1.sources.r1.type = TAILDIR
a1.sources.r1.channels = c1
a1.sources.r1.positionFile = /var/log/flume/taildir_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /var/log/test1/*.log
a1.sources.ri.maxBatchCount = 1000 
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.sinks.k1.type = logger 
a1.sinks.k1.channel = c1

and start flume with it

//代码占位符
bin/flume-ng agent --conf conf --conf-file conf/example.conf --name a1 -Dflume.root.logger=INFO,console 

then ctrl + c stop it.

Just wait a moment, and then start flume again, then it will throw an exception like below.

//代码占位符
ERROR [lifecycleSupervisor-1-1] (org.apache.flume.source.taildir.ReliableTaildirEventReader.loadPositionFile:147)  - Failed loading positionFile: /var/log/flume/taildir_position.json
java.io.EOFException: End of input at line 1 column 1
        at com.google.gson.stream.JsonReader.nextNonWhitespace(JsonReader.java:954) ~[gson-2.2.2.jar:?]
        at com.google.gson.stream.JsonReader.nextValue(JsonReader.java:771) ~[gson-2.2.2.jar:?]
        at com.google.gson.stream.JsonReader.peek(JsonReader.java:367) ~[gson-2.2.2.jar:?]
        at com.google.gson.stream.JsonReader.expect(JsonReader.java:337) ~[gson-2.2.2.jar:?]
        at com.google.gson.stream.JsonReader.beginArray(JsonReader.java:306) ~[gson-2.2.2.jar:?]
        at org.apache.flume.source.taildir.ReliableTaildirEventReader.loadPositionFile(ReliableTaildirEventReader.java:111) [flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
        at org.apache.flume.source.taildir.ReliableTaildirEventReader.<init>(ReliableTaildirEventReader.java:96) [flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
        at org.apache.flume.source.taildir.ReliableTaildirEventReader.<init>(ReliableTaildirEventReader.java:49) [flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
        at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:355) [flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
        at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:105) [flume-taildir-source-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
        at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:71) [flume-ng-core-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249) [flume-ng-core-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0-292]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0-292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0-292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0-292]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0-292]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0-292]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0-292] 

Fix

We just add a condition in ReliableTaildirEventReader constructor.

The pr/patch will be submitted as as shown below.

Or let it still exist and using warning log instead of exception thrown is better?

 

Issue Links

• links to: GitHub Pull Request #367