Flume
  1. Flume
  2. FLUME-1535

Ability to specify the capacity of MemoryChannel in bytes

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.3.0
    • Component/s: Channel
    • Labels:

      Description

      Users should be able to specify the capacity of memory channel in bytes. Currently users are able to specify the capacity in terms of max number of events. This causes issues when the size of the events can vary significantly (a couple of KB to MB) and could easily result in the Flume agent running into OOM.

      1. FLUME-1535-0.patch
        11 kB
        Ted Malaska
      2. FLUME-1535-1.patch
        12 kB
        Ted Malaska
      3. FLUME-1535-4.patch
        13 kB
        Ted Malaska
      4. FLUME-1535-5-Atomic.patch
        12 kB
        Ted Malaska
      5. FLUME-1535-Semaphore-0.patch
        14 kB
        Ted Malaska
      6. FLUME-1535-Semaphore-1.patch
        15 kB
        Ted Malaska
      7. FLUME-1535-Semaphore-3.patch
        14 kB
        Ted Malaska

        Issue Links

          Activity

          Hide
          Ted Malaska added a comment -

          Just noting a comment by Mike P that should be remembered when reviewing this patch

          "One possibly tricky issue is incrementing the memory usage on put (but undoing it on rollback) and only decrementing it on commit of takes."

          Show
          Ted Malaska added a comment - Just noting a comment by Mike P that should be remembered when reviewing this patch "One possibly tricky issue is incrementing the memory usage on put (but undoing it on rollback) and only decrementing it on commit of takes."
          Hide
          Ted Malaska added a comment - - edited

          Not totally finished.

          But this patch contains a working implementation and junits.

          Please feel free to comment.

          Tomorrow I plan to give it one more look over and add some comments and exit some wording.

          Show
          Ted Malaska added a comment - - edited Not totally finished. But this patch contains a working implementation and junits. Please feel free to comment. Tomorrow I plan to give it one more look over and add some comments and exit some wording.
          Hide
          Ted Malaska added a comment -

          Did some clean up.

          This is a fulling working and tested patch.

          Show
          Ted Malaska added a comment - Did some clean up. This is a fulling working and tested patch.
          Hide
          Ted Malaska added a comment -

          How it works:

          1. The user will be able to define a byteCapacity and a byteCapacityBufferPercentage.
          2. Events byte size will be estimated from there body contents
          3. On put bytes are added to current total
          4. On commit any uncommitted takes are removed from the current byte total
          5. On rollover any uncommitted puts are removed from the current byte total

          Show
          Ted Malaska added a comment - How it works: 1. The user will be able to define a byteCapacity and a byteCapacityBufferPercentage. 2. Events byte size will be estimated from there body contents 3. On put bytes are added to current total 4. On commit any uncommitted takes are removed from the current byte total 5. On rollover any uncommitted puts are removed from the current byte total
          Hide
          Ted Malaska added a comment -

          Thank you Mike for the review.

          Summary of my changes
          -Changed currentByteUsage to AtomicLong
          -Also added synchronized(currentByteUsage) to put.

          Show
          Ted Malaska added a comment - Thank you Mike for the review. Summary of my changes -Changed currentByteUsage to AtomicLong -Also added synchronized(currentByteUsage) to put.
          Hide
          Ted Malaska added a comment -

          here is the solution with AtomicLong. Tomorrow I will look at a possible solution using a Semaphore.

          Let me know if there are any problems with the AtomicLong solution and how might the use of Semaphore help.

          Show
          Ted Malaska added a comment - here is the solution with AtomicLong. Tomorrow I will look at a possible solution using a Semaphore. Let me know if there are any problems with the AtomicLong solution and how might the use of Semaphore help.
          Hide
          Ted Malaska added a comment -

          Implemented byte capacity with semaphores

          Show
          Ted Malaska added a comment - Implemented byte capacity with semaphores
          Hide
          Ted Malaska added a comment -

          Added another unit test

          Show
          Ted Malaska added a comment - Added another unit test
          Hide
          Ted Malaska added a comment -

          Fixed multi config change issue.

          Also added unit test to check for such an error in the future.

          Show
          Ted Malaska added a comment - Fixed multi config change issue. Also added unit test to check for such an error in the future.
          Hide
          Hari Shreedharan added a comment -

          Excellent work. Thanks for pulling it through. Patch committed, rev: 246c1b0213a2089e6cb684d1d982cd03015fb09e. Thanks Ted!

          Show
          Hari Shreedharan added a comment - Excellent work. Thanks for pulling it through. Patch committed, rev: 246c1b0213a2089e6cb684d1d982cd03015fb09e. Thanks Ted!
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #312 (See https://builds.apache.org/job/flume-trunk/312/)
          FLUME-1535. Ability to specify the capacity of MemoryChannel in bytes. (Revision 246c1b0213a2089e6cb684d1d982cd03015fb09e)

          Result = SUCCESS
          hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=246c1b0213a2089e6cb684d1d982cd03015fb09e
          Files :

          • flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java
          • flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java
          Show
          Hudson added a comment - Integrated in flume-trunk #312 (See https://builds.apache.org/job/flume-trunk/312/ ) FLUME-1535 . Ability to specify the capacity of MemoryChannel in bytes. (Revision 246c1b0213a2089e6cb684d1d982cd03015fb09e) Result = SUCCESS hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=246c1b0213a2089e6cb684d1d982cd03015fb09e Files : flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java

            People

            • Assignee:
              Ted Malaska
              Reporter:
              Jayant Shekhar
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development