Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2801

Performance improvement on TailDir source

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v1.7.0
    • Fix Version/s: v1.7.0
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Flags:
      Patch

      Description

      This a proposal of performance improvement for new tailing source FLUME-2498.
      Taildir source reads a file by 1byte, so the performance is very low compared to tailing on exec source.
      I tested lot's of ways to improve performance and implemented the best one.

      Changes.

      • Reading a file by a 8k block instead of 1 byte.
      • Use byte[] for handling data instead of ByteArrayDataOutput/ByteBuffer(direct)/.. for better performance.
      • Don't convert byte[] to string and vice verse.

      Simple file reading test results.

      File size: 100 MB,
      Line size: 500 byte

      Estimated time to read the file:

      Reading 1byte(Using the code in Taildir) 32544 ms
      Reading 8K Block 431 ms

      Testing on flume, it catches up the performance of tailing on exec source. (30x performance boost)

      1. FLUME-2801.patch
        9 kB
        Jun Seok Hong
      2. FLUME-2801-1.patch
        9 kB
        Jun Seok Hong
      3. FLUME-2801-2.patch
        9 kB
        Satoshi Iijima

        Activity

        Hide
        siefried12 Jun Seok Hong added a comment -

        Fixed a bug related to testBackoffWithoutNewLine().

        Show
        siefried12 Jun Seok Hong added a comment - Fixed a bug related to testBackoffWithoutNewLine().
        Hide
        roshan_naik Roshan Naik added a comment -

        Satoshi Iijima could you help review this patch ?

        Show
        roshan_naik Roshan Naik added a comment - Satoshi Iijima could you help review this patch ?
        Hide
        iijima_satoshi Satoshi Iijima added a comment -

        Sorry for not responsing sooner.
        +1. I attached a patch added a minor refactoring.

        Show
        iijima_satoshi Satoshi Iijima added a comment - Sorry for not responsing sooner. +1. I attached a patch added a minor refactoring.
        Hide
        roshan_naik Roshan Naik added a comment - - edited

        +1
        Thanks Satoshi Iijima for the review. Im running tests.. will commit soon.

        Show
        roshan_naik Roshan Naik added a comment - - edited +1 Thanks Satoshi Iijima for the review. Im running tests.. will commit soon.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 0421fa2ab1eb9575b34bbb2f44e8c6d83842eaeb in flume's branch refs/heads/trunk from Roshan Naik
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=0421fa2 ]

        FLUME-2801. Performance improvement on TailDir source

        (Jun Seok Hong via Satoshi Iijima and Roshan Naik)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 0421fa2ab1eb9575b34bbb2f44e8c6d83842eaeb in flume's branch refs/heads/trunk from Roshan Naik [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=0421fa2 ] FLUME-2801 . Performance improvement on TailDir source (Jun Seok Hong via Satoshi Iijima and Roshan Naik)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 9156c36608dad0c68679c89f15280fa189a30d86 in flume's branch refs/heads/flume-1.7 from Roshan Naik
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9156c36 ]

        FLUME-2801. Performance improvement on TailDir source

        (Jun Seok Hong via Satoshi Iijima and Roshan Naik)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 9156c36608dad0c68679c89f15280fa189a30d86 in flume's branch refs/heads/flume-1.7 from Roshan Naik [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9156c36 ] FLUME-2801 . Performance improvement on TailDir source (Jun Seok Hong via Satoshi Iijima and Roshan Naik)
        Hide
        roshan_naik Roshan Naik added a comment -

        Commited

        Thanks for this patch Jun Seok Hong. I have added you to the contributors list, so you can assign open jiras to yourself.

        Satoshi Iijima .. i have noted u as a reviewer in the commit.

        Show
        roshan_naik Roshan Naik added a comment - Commited Thanks for this patch Jun Seok Hong . I have added you to the contributors list, so you can assign open jiras to yourself. Satoshi Iijima .. i have noted u as a reviewer in the commit.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Flume-trunk-hbase-1 #144 (See https://builds.apache.org/job/Flume-trunk-hbase-1/144/)
        FLUME-2801. Performance improvement on TailDir source (roshan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=0421fa2ab1eb9575b34bbb2f44e8c6d83842eaeb)

        • flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TailFile.java
        • flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-1 #144 (See https://builds.apache.org/job/Flume-trunk-hbase-1/144/ ) FLUME-2801 . Performance improvement on TailDir source (roshan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=0421fa2ab1eb9575b34bbb2f44e8c6d83842eaeb ) flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/TailFile.java flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java

          People

          • Assignee:
            siefried12 Jun Seok Hong
            Reporter:
            siefried12 Jun Seok Hong
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development