Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3101

taildir source may endless loop when tail a file

    XMLWordPrintableJSON

Details

    Description

      If there are many files in the path that need to be tail, and there is a file written by high frequency (for example, there are file a, file b and file c in the path, file a is written at high frequency), taildir can read the batchSize size event from file a everytime, then taildir will only read data from file a, other files will not to be read, because in TaildirSource.tailFileProcess will into an endless loop.

      code:

      TaildirSource.java
      private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
          throws IOException, InterruptedException {
        while (true) {
          // if events.size >= batchSize will not break while, 
          // then into endless loop to only read tf
          if (events.size() < batchSize) {
            break;
          }
        }
      }
      

      Attachments

        1. FLUME-3101-3.patch
          6 kB
          hunshenshi
        2. FLUME-3101-2.patch
          6 kB
          hunshenshi
        3. FLUME-3101-1.patch
          3 kB
          hunshenshi
        4. FLUME-3101-0.patch
          1 kB
          hunshenshi

        Issue Links

          Activity

            People

              turcsanyip Peter Turcsanyi
              hunhun hunshenshi
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: