Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Not A Problem
    • Affects Version/s: v1.2.0
    • Fix Version/s: None
    • Component/s: Channel
    • Labels:

      Description

      The performance result of FileChannel is below expectation, almost 5MB/s. May be it should be tune to reduce effect of ingest system.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          158d 22h 35m 1 Hari Shreedharan 09/Jan/13 02:15
          Mike Percy made changes -
          Fix Version/s v1.4.0 [ 12323372 ]
          Hari Shreedharan made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Not A Problem [ 8 ]
          Hide
          Hari Shreedharan added a comment -

          As discussed here, removing the fsync will kill the channel's reliability semantics, so closing this as Not A Problem

          Show
          Hari Shreedharan added a comment - As discussed here, removing the fsync will kill the channel's reliability semantics, so closing this as Not A Problem
          Hide
          Juhani Connolly added a comment -

          I think this issue should be closed. The "fix" involved a loss of guarrantees made by the file channel.

          As is, the recommended methods for getting good performance are large batches, and separating disks to allow for sequential writes. So long as each batch is flushed before commit, there aren't going to be any silver bullets that will result in huge performance boosts.

          Show
          Juhani Connolly added a comment - I think this issue should be closed. The "fix" involved a loss of guarrantees made by the file channel. As is, the recommended methods for getting good performance are large batches, and separating disks to allow for sequential writes. So long as each batch is flushed before commit, there aren't going to be any silver bullets that will result in huge performance boosts.
          Brock Noland made changes -
          Fix Version/s v1.4.0 [ 12323372 ]
          Fix Version/s v1.3.0 [ 12322140 ]
          Denny Ye made changes -
          Remote Link This issue links to "Review Request (Web Link)" [ 10503 ]
          Denny Ye made changes -
          Field Original Value New Value
          Attachment FLUME-1423.patch [ 12539002 ]
          Hide
          Denny Ye added a comment -

          Patrick, I forwarded the mail-list to you. It contains testing environment, configuration. Only difference between file and memory channel benchmark is channel type. There is single Flume process hold local disk.

          Show
          Denny Ye added a comment - Patrick, I forwarded the mail-list to you. It contains testing environment, configuration. Only difference between file and memory channel benchmark is channel type. There is single Flume process hold local disk.
          Hide
          NO NAME added a comment -

          Danny - can you give more information about the environment you are running on?

          In particular:

          Can you test the throughout of the exact same setup with an in-memory channel, and give the resulting throughput, to ensure that this is indeed the bottleneck?

          What are the access patterns of the disk that the FileChannel is using? Is it a dedicated disk? Or are there other processes reading/writing against the disk?

          Can you give the configuration of the agent, including the sources and information about any batching taking place within those sources?

          Thanks,
          Patrick

          Show
          NO NAME added a comment - Danny - can you give more information about the environment you are running on? In particular: Can you test the throughout of the exact same setup with an in-memory channel, and give the resulting throughput, to ensure that this is indeed the bottleneck? What are the access patterns of the disk that the FileChannel is using? Is it a dedicated disk? Or are there other processes reading/writing against the disk? Can you give the configuration of the agent, including the sources and information about any batching taking place within those sources? Thanks, Patrick
          Denny Ye created issue -

            People

            • Assignee:
              Denny Ye
              Reporter:
              Denny Ye
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development