Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3101

taildir source may endless loop when tail a file

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      If there are many files in the path that need to be tail, and there is a file written by high frequency (for example, there are file a, file b and file c in the path, file a is written at high frequency), taildir can read the batchSize size event from file a everytime, then taildir will only read data from file a, other files will not to be read, because in TaildirSource.tailFileProcess will into an endless loop.

      code:

      TaildirSource.java
      private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
          throws IOException, InterruptedException {
        while (true) {
          // if events.size >= batchSize will not break while, 
          // then into endless loop to only read tf
          if (events.size() < batchSize) {
            break;
          }
        }
      }
      

        Attachments

        1. FLUME-3101-3.patch
          6 kB
          hunshenshi
        2. FLUME-3101-2.patch
          6 kB
          hunshenshi
        3. FLUME-3101-1.patch
          3 kB
          hunshenshi
        4. FLUME-3101-0.patch
          1 kB
          hunshenshi

          Activity

            People

            • Assignee:
              turcsanyip Peter Turcsanyi
              Reporter:
              hunhun hunshenshi

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment