[FLUME-2180] SpoolDirectorySource performs poorly when there are thousands of files

• Type: Improvement
• Status: Patch Available
• Priority: Trivial
• Resolution: Unresolved
• Affects Version/s: 1.4.0
• Component/s: Sinks+Sources
• Labels: patch performance

Environment

Flume 1.4.0 running on Ubuntu 12.04.2 LTS x86_64

• Assignee: Unassigned
• Reporter: Mike Hobbs
• Created: 29/Aug/13 19:00
• Updated: 29/Aug/13 19:12

Description

org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile() spikes the CPU when there are thousands of files in the spoolDirectory. So much so that the source was unable to keep up and even more files started accumulating.