Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2344

New source for tailing files

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 1.4.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      Centos 6.4, Java 1.6.0_34

    • Release Note:
      Hide
      Patch has been uploaded.
      It has been create a new source for tailing files in Flume.
      If a file rotates more than once, you couldn't recover those data with this Source.
      For example, if you're watching catalina.out and this file rotates to catalina.out.1 and Flume is down, when you turn Flume on, you read the data that you didn't read in catalina.out.1, but if while Flume is down there was another rotation (catalina.out.2), you couldn't recover those data.
      If for some reason it's not enable to recover data, the source'll start to read the original file from the beginning.

      To detect rotation and which file has to read, the Source keeps a file with file is watching, last offset we read and last time the rotated file was modified.
      Show
      Patch has been uploaded. It has been create a new source for tailing files in Flume. If a file rotates more than once, you couldn't recover those data with this Source. For example, if you're watching catalina.out and this file rotates to catalina.out.1 and Flume is down, when you turn Flume on, you read the data that you didn't read in catalina.out.1, but if while Flume is down there was another rotation (catalina.out.2), you couldn't recover those data. If for some reason it's not enable to recover data, the source'll start to read the original file from the beginning. To detect rotation and which file has to read, the Source keeps a file with file is watching, last offset we read and last time the rotated file was modified.

      Description

      New source to be able to tail a file. There's a extra file when it saves the last offset until it has been read. So, it Flume is down, it could read the data it lost. It has been implemented control about rotated files.

      The possible variables to configure this source are:
      -BufferSize gives us the possibility to send little to little.
      -Separator to cut the lines wherever we want to to generate our events.
      -WatchedFile to indicate what file we want to watch.
      -RotatedFile to indicate where the wacthed file is going to rotate.
      -Type of events to indicate if we generate one event per line or we want to group many lines and just emit one event.

        Attachments

        1. FLUME-2344-0.patch
          33 kB
          Guillermo Ortiz Fernández, Pragsis.

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                guillermo.of Guillermo Ortiz Fernández, Pragsis.
              • Votes:
                3 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: