[FLUME-3101] taildir source may endless loop when tail a file - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 1.7.0
Fix Version/s: 1.9.0
Component/s: Sinks+Sources
Labels:
- patch
- taildirsource

Description

If there are many files in the path that need to be tail, and there is a file written by high frequency (for example, there are file a, file b and file c in the path, file a is written at high frequency), taildir can read the batchSize size event from file a everytime, then taildir will only read data from file a, other files will not to be read, because in TaildirSource.tailFileProcess will into an endless loop.

code:

TaildirSource.java

private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
    throws IOException, InterruptedException {
  while (true) {
    // if events.size >= batchSize will not break while, 
    // then into endless loop to only read tf
    if (events.size() < batchSize) {
      break;
    }
  }
}