Flume
  1. Flume
  2. FLUME-457

Persist some tails state to allow nodes to "continue" from previous positions.

    Details

    • Type: New Feature New Feature
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v0.9.4
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      This is likely not something that will be completely robust it is an oft requested feature.

        Activity

        Hide
        Thomas Rega added a comment -

        Will do. I am going to modify the changes first and add some tests, then I will be able to submit the changes for serious review. Thanks Otis.

        Show
        Thomas Rega added a comment - Will do. I am going to modify the changes first and add some tests, then I will be able to submit the changes for serious review. Thanks Otis.
        Hide
        Otis Gospodnetic added a comment - - edited

        Thomas - great to hear this!
        2 suggestions:

        1) leave a comment here describing your observations about how well this works for you (in production?), where/when/if it breaks, if you have thoughts on how one might be able to improve the current weak spots
        2) attach a patch with your crude implementation and be prepared to iterate on it based on other Flume developers' feedback

        Thanks!

        Show
        Otis Gospodnetic added a comment - - edited Thomas - great to hear this! 2 suggestions: 1) leave a comment here describing your observations about how well this works for you (in production?), where/when/if it breaks, if you have thoughts on how one might be able to improve the current weak spots 2) attach a patch with your crude implementation and be prepared to iterate on it based on other Flume developers' feedback Thanks!
        Hide
        Thomas Rega added a comment - - edited

        I have written a crude implementation of this feature by modifying the taildirsource and cursor objects so they store positional data of what is being read in XML files in the logdir. Basically, the position of the cursor on the RAF, the last modified DT (stored as a long), and the file size are being stored so one can recover after the flume agent restarts.

        Does anyone have any suggestions on how I could change this implementation? I am almost certian that in it's current form, my crude implementation would be rejected.

        Show
        Thomas Rega added a comment - - edited I have written a crude implementation of this feature by modifying the taildirsource and cursor objects so they store positional data of what is being read in XML files in the logdir. Basically, the position of the cursor on the RAF, the last modified DT (stored as a long), and the file size are being stored so one can recover after the flume agent restarts. Does anyone have any suggestions on how I could change this implementation? I am almost certian that in it's current form, my crude implementation would be rejected.
        Hide
        Otis Gospodnetic added a comment - - edited

        From https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/c02cd92b2c8fbbcf :

        Shouldn't it be possible to configure Flume Agent to "go check up to N of those other (older/rotated/compressed) files for data in xyz dir if the current log file doesn't seem to be the one you were reading just before you went down"?

        2 "optimizations":

        • If Flume's tail kept track of the last line it processed in a file then it could go scan old files, find the point where it stopped tailing it, and read from there until the end of the file. And do so for up to N files (or maybe all files less than H minutes old).
        • Also, if there is a way to "jump" to the very last line of a file (or at least almost very last line), then if Flume's tail kept a record of the very last line of each file that it processed before it switched to a new file (because of log rotation), it could quickly tell if the file was completely read before or not (and if not, it could do the scan as described above).
        Show
        Otis Gospodnetic added a comment - - edited From https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/c02cd92b2c8fbbcf : Shouldn't it be possible to configure Flume Agent to "go check up to N of those other (older/rotated/compressed) files for data in xyz dir if the current log file doesn't seem to be the one you were reading just before you went down"? 2 "optimizations": If Flume's tail kept track of the last line it processed in a file then it could go scan old files, find the point where it stopped tailing it, and read from there until the end of the file. And do so for up to N files (or maybe all files less than H minutes old). Also, if there is a way to "jump" to the very last line of a file (or at least almost very last line), then if Flume's tail kept a record of the very last line of each file that it processed before it switched to a new file (because of log rotation), it could quickly tell if the file was completely read before or not (and if not, it could do the scan as described above).
        Hide
        Nicholas Verbeck added a comment -

        I don't know if this would be best to create a new source that inherits from tail to allow for persistence or enhance tail to have another arg for this. But I can imagine if you record the inode id and position of read to disk you should be able to survive a restart of the node as well as a log rotate while the node is down.

        Show
        Nicholas Verbeck added a comment - I don't know if this would be best to create a new source that inherits from tail to allow for persistence or enhance tail to have another arg for this. But I can imagine if you record the inode id and position of read to disk you should be able to survive a restart of the node as well as a log rotate while the node is down.

          People

          • Assignee:
            Unassigned
            Reporter:
            Jonathan Hsieh
          • Votes:
            4 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development