Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
We are trying to push number of log files are located at folder "/path-to-logs/processed_logs/" through flume to kafka.
Now when flume agent is started it tries to send a few number of files but after that gives the below error:
And finally, it is unable to tail the remaining files. This inhibits the further flow of data.
[ERROR - org.apache.flume.source.taildir.TaildirSource.writePosition(TaildirSource.java:334)] Failed writing positionFile
java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.google.common.collect.ImmutableMap.of(ImmutableMap.java:95)
at org.apache.flume.source.taildir.TaildirSource.toPosInfoJson(TaildirSource.java:349)
at org.apache.flume.source.taildir.TaildirSource.writePosition(TaildirSource.java:330)
at org.apache.flume.source.taildir.TaildirSource.access$600(TaildirSource.java:59)
at org.apache.flume.source.taildir.TaildirSource$PositionWriterRunnable.run(TaildirSource.java:320)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
[ERROR - org.apache.flume.source.taildir.TaildirSource.process(TaildirSource.java:236)] Unable to tail files
java.lang.OutOfMemoryError: GC overhead limit exceeded