Commons IO
  1. Commons IO
  2. IO-278

Improve Tailer performance with buffered reads

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.4
    • Component/s: None
    • Labels:
      None

      Description

      I noticed Tailer read performances are pretty poor when dealing with large, frequently written, log files, and this is due to the use of RandomAccessFile which does unbuffered reads, hence causing lots of disk I/O.

      So I improved the Tailer implementation by introducing buffered reads: it works by loading large (configurable) file chunks in memory, and reading lines from there; this enhances performances in my tests from 10x to 30x depending on the file size.
      I also added two test cases: one to simulate reading of a large file (you can use it to compare performances), the other to verify correct handling on buffer breaks; obviously, all tests pass.

      I'm attaching the diff files, let me know if it's okay for you guys!

      1. Tailer.diff
        10 kB
        Sergio Bossa
      2. TailerTest.diff
        7 kB
        Sergio Bossa

        Issue Links

          Activity

          Hide
          Sergio Bossa added a comment -

          Also, while I was at it I fixed a concurrency bug in TestTailerListener causing intermittent test failures (the lines list needed to be thread safe).

          Show
          Sergio Bossa added a comment - Also, while I was at it I fixed a concurrency bug in TestTailerListener causing intermittent test failures (the lines list needed to be thread safe).
          Hide
          James Herrmann added a comment -

          Hi Sergio,

          Have you heard if these last two fixes (IO-278/IO-279) under evaluation for inclusion into the official build?

          Thanks,

          Jim

          Show
          James Herrmann added a comment - Hi Sergio, Have you heard if these last two fixes ( IO-278 / IO-279 ) under evaluation for inclusion into the official build? Thanks, Jim
          Hide
          Sergio Bossa added a comment -

          Hi Jim,

          Commons-IO team doesn't seem to be much responsive, so I had to create a fork from that at: https://github.com/sbtourist/tayler

          Feel free to give it a look.

          Show
          Sergio Bossa added a comment - Hi Jim, Commons-IO team doesn't seem to be much responsive, so I had to create a fork from that at: https://github.com/sbtourist/tayler Feel free to give it a look.
          Hide
          Gary Gregory added a comment -

          Hello,

          Can you create your patches on top of SVN trunk? I'll take a look this week.

          Thank you,
          Gary

          Show
          Gary Gregory added a comment - Hello, Can you create your patches on top of SVN trunk? I'll take a look this week. Thank you, Gary
          Hide
          James Herrmann added a comment - - edited

          Looks like theses patches have been rolled in. So, congrats to all on that. I went ahead and maven repo'ed your fork anyway because of the IO-279 issue. Tailer would read from start of file on frequent occasions. Not good when you're sending alerts! Sergio, does your fork handle the IO-269 bug? Right now, that's the last known issue I'm concerned about.

          Thanks!

          Jim

          Edited: wrong bug #

          Show
          James Herrmann added a comment - - edited Looks like theses patches have been rolled in. So, congrats to all on that. I went ahead and maven repo'ed your fork anyway because of the IO-279 issue. Tailer would read from start of file on frequent occasions. Not good when you're sending alerts! Sergio, does your fork handle the IO-269 bug? Right now, that's the last known issue I'm concerned about. Thanks! Jim Edited: wrong bug #
          Hide
          Sergio Bossa added a comment -

          Hi James, I think this is not the right place to discuss issues related to my fork, please use Tayler github tracker for that.

          Thanks,

          Sergio B.

          Show
          Sergio Bossa added a comment - Hi James, I think this is not the right place to discuss issues related to my fork, please use Tayler github tracker for that. Thanks, Sergio B.
          Hide
          Sebb added a comment -

          Buffering was added as part of IO-332.
          Added tests.

          Show
          Sebb added a comment - Buffering was added as part of IO-332 . Added tests.

            People

            • Assignee:
              Unassigned
              Reporter:
              Sergio Bossa
            • Votes:
              3 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development