Flume
  1. Flume
  2. FLUME-252

Update Tail to get rid of races and truncation problems.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: v0.9.0, v0.9.1
    • Fix Version/s: v0.9.2
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      The first tail implementation used buffered readers and file readers. This caused problems because the read call was blocking and couldn't shutdown properly.

      The second tail implementation was mroe closely based on gnu tail's C implementation but relies on RandomAccessFile. This version had problems with races (restarting from beginning FLUME-218) and truncation (new test in FLUME-218 patch by Eric Sammer).

      The new approach will likely use NIO and nonblocking IO to act cleanly, or possibly use a JNI based approach to get to unix system calls to get and follow file descriptors or inode numbers.

        Issue Links

          Activity

          Jonathan Hsieh created issue -
          Jonathan Hsieh made changes -
          Field Original Value New Value
          Link This issue relates to FLUME-218 [ FLUME-218 ]
          Jonathan Hsieh made changes -
          Link This issue relates to FLUME-205 [ FLUME-205 ]
          Jonathan Hsieh made changes -
          Link This issue relates to FLUME-248 [ FLUME-248 ]
          Jonathan Hsieh made changes -
          Link This issue relates to FLUME-148 [ FLUME-148 ]
          Jonathan Hsieh made changes -
          Assignee Jonathan Hsieh [ jmhsieh ]
          Jonathan Hsieh made changes -
          Affects Version/s v0.9.0 [ 10014 ]
          Affects Version/s v0.9.1 [ 10013 ]
          Description The first tail implementation used buffered readers and file readers. This caused problems because the read call was blocking and couldn't shutdown properly.

          The second tail implementation was mroe closely based on gnu tail's C implementation but relies on RandomAccessFile. This version had problems with races (restarting from beginning FLUME-218) and truncation (new test in FLUME-218 patch by Eric Sammer).

          The new approach will likely use NIO and nonblocking IO to cleanly, or possible a JNI based approach to get to unix system calls to get and follow file descriptors or inode numbers.
          The first tail implementation used buffered readers and file readers. This caused problems because the read call was blocking and couldn't shutdown properly.

          The second tail implementation was mroe closely based on gnu tail's C implementation but relies on RandomAccessFile. This version had problems with races (restarting from beginning FLUME-218) and truncation (new test in FLUME-218 patch by Eric Sammer).

          The new approach will likely use NIO and nonblocking IO to act cleanly, or possibly use a JNI based approach to get to unix system calls to get and follow file descriptors or inode numbers.
          Jonathan Hsieh made changes -
          Component/s Sinks+Sources [ 10041 ]
          Jonathan Hsieh made changes -
          Link This issue relates to FLUME-232 [ FLUME-232 ]
          Jonathan Hsieh made changes -
          Status Open [ 1 ] Patch Available [ 10000 ]
          Jonathan Hsieh made changes -
          Attachment 0001-FLUME-252-Update-tail-to-get-rid-of-races-and-trunca.patch [ 10261 ]
          Attachment License Granted license to ASF [ licensed ]
          Jonathan Hsieh made changes -
          Link This issue blocks FLUME-258 [ FLUME-258 ]
          Jonathan Hsieh made changes -
          Fix Version/s v0.9.2 [ 10022 ]
          Jonathan Hsieh made changes -
          Link This issue relates to FLUME-320 [ FLUME-320 ]
          Jonathan Hsieh made changes -
          Status Patch Available [ 10000 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Jonathan Hsieh made changes -
          Link This issue blocks FLUME-323 [ FLUME-323 ]
          Jonathan Hsieh made changes -
          Link This issue blocks FLUME-323 [ FLUME-323 ]
          Jonathan Hsieh made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Jonathan Hsieh made changes -
          Link This issue is duplicated by FLUME-518 [ FLUME-518 ]
          Mark Thomas made changes -
          Project Import Tue Aug 02 16:57:12 UTC 2011 [ 1312304232406 ]
          hailinzeng made changes -
          Link This issue relates to FLUME-2354 [ FLUME-2354 ]

            People

            • Assignee:
              Jonathan Hsieh
              Reporter:
              Jonathan Hsieh
            • Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development