Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Not a Problem
    • Affects Version/s: v1.2.0
    • Fix Version/s: None
    • Component/s: Channel
    • Labels:

      Description

      The performance result of FileChannel is below expectation, almost 5MB/s. May be it should be tune to reduce effect of ingest system.

        Issue Links

          Activity

          Hide
          NO NAME added a comment -

          Danny - can you give more information about the environment you are running on?

          In particular:

          Can you test the throughout of the exact same setup with an in-memory channel, and give the resulting throughput, to ensure that this is indeed the bottleneck?

          What are the access patterns of the disk that the FileChannel is using? Is it a dedicated disk? Or are there other processes reading/writing against the disk?

          Can you give the configuration of the agent, including the sources and information about any batching taking place within those sources?

          Thanks,
          Patrick

          Show
          NO NAME added a comment - Danny - can you give more information about the environment you are running on? In particular: Can you test the throughout of the exact same setup with an in-memory channel, and give the resulting throughput, to ensure that this is indeed the bottleneck? What are the access patterns of the disk that the FileChannel is using? Is it a dedicated disk? Or are there other processes reading/writing against the disk? Can you give the configuration of the agent, including the sources and information about any batching taking place within those sources? Thanks, Patrick
          Hide
          Denny Ye added a comment -

          Patrick, I forwarded the mail-list to you. It contains testing environment, configuration. Only difference between file and memory channel benchmark is channel type. There is single Flume process hold local disk.

          Show
          Denny Ye added a comment - Patrick, I forwarded the mail-list to you. It contains testing environment, configuration. Only difference between file and memory channel benchmark is channel type. There is single Flume process hold local disk.
          Hide
          Juhani Connolly added a comment -

          I think this issue should be closed. The "fix" involved a loss of guarrantees made by the file channel.

          As is, the recommended methods for getting good performance are large batches, and separating disks to allow for sequential writes. So long as each batch is flushed before commit, there aren't going to be any silver bullets that will result in huge performance boosts.

          Show
          Juhani Connolly added a comment - I think this issue should be closed. The "fix" involved a loss of guarrantees made by the file channel. As is, the recommended methods for getting good performance are large batches, and separating disks to allow for sequential writes. So long as each batch is flushed before commit, there aren't going to be any silver bullets that will result in huge performance boosts.
          Hide
          Hari Shreedharan added a comment -

          As discussed here, removing the fsync will kill the channel's reliability semantics, so closing this as Not A Problem

          Show
          Hari Shreedharan added a comment - As discussed here, removing the fsync will kill the channel's reliability semantics, so closing this as Not A Problem

            People

            • Assignee:
              Denny Ye
              Reporter:
              Denny Ye
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development