Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3101

taildir source may endless loop when tail a file

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      If there are many files in the path that need to be tail, and there is a file written by high frequency (for example, there are file a, file b and file c in the path, file a is written at high frequency), taildir can read the batchSize size event from file a everytime, then taildir will only read data from file a, other files will not to be read, because in TaildirSource.tailFileProcess will into an endless loop.

      code:

      TaildirSource.java
      private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
          throws IOException, InterruptedException {
        while (true) {
          // if events.size >= batchSize will not break while, 
          // then into endless loop to only read tf
          if (events.size() < batchSize) {
            break;
          }
        }
      }
      

      Attachments

        1. FLUME-3101-3.patch
          6 kB
          hunshenshi
        2. FLUME-3101-2.patch
          6 kB
          hunshenshi
        3. FLUME-3101-1.patch
          3 kB
          hunshenshi
        4. FLUME-3101-0.patch
          1 kB
          hunshenshi

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            turcsanyip Peter Turcsanyi
            hunhun hunshenshi
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment