[FLUME-3101] taildir source may endless loop when tail a file

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.0
• Fix Version/s: 1.9.0
• Component/s: Sinks+Sources
• Labels: patch taildirsource
• Assignee: Peter Turcsanyi
• Reporter: hunshenshi
• Created: 01/Jun/17 10:13
• Updated: 22/Nov/18 21:00
• Resolved: 22/Nov/18 21:00

Description

If there are many files in the path that need to be tail, and there is a file written by high frequency (for example, there are file a, file b and file c in the path, file a is written at high frequency), taildir can read the batchSize size event from file a everytime, then taildir will only read data from file a, other files will not to be read, because in TaildirSource.tailFileProcess will into an endless loop.

code:

TaildirSource.java

private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
    throws IOException, InterruptedException {
  while (true) {
    // if events.size >= batchSize will not break while, 
    // then into endless loop to only read tf
    if (events.size() < batchSize) {
      break;
    }
  }
}

Issue Links

• relates to: FLUME-3294 Fix polling logic in TaildirSource
• links to: GitHub Pull Request #240
• links to: ReviewRequest_FLUME-3101