Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3101

taildir source may endless loop when tail a file

    Details

      Description

      If there are many files in the path that need to be tail, and there is a file written by high frequency (for example, there are file a, file b and file c in the path, file a is written at high frequency), taildir can read the batchSize size event from file a everytime, then taildir will only read data from file a, other files will not to be read, because in TaildirSource.tailFileProcess will into an endless loop.

      code:

      TaildirSource.java
      private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
          throws IOException, InterruptedException {
        while (true) {
          // if events.size >= batchSize will not break while, 
          // then into endless loop to only read tf
          if (events.size() < batchSize) {
            break;
          }
        }
      }
      

        Attachments

        1. FLUME-3101-0.patch
          1 kB
          hunshenshi
        2. FLUME-3101-1.patch
          3 kB
          hunshenshi
        3. FLUME-3101-2.patch
          6 kB
          hunshenshi
        4. FLUME-3101-3.patch
          6 kB
          hunshenshi

          Issue Links

            Activity

              People

              • Assignee:
                turcsanyip Peter Turcsanyi
                Reporter:
                hunhun hunshenshi
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: