Flume
  1. Flume
  2. FLUME-1621

Document new MemoryChannel parameters in Flume User Guide

    Details

    • Type: Documentation Documentation
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.3.0
    • Fix Version/s: v1.4.0
    • Component/s: Channel
    • Labels:
      None

      Description

      New parameters were added in FLUME-1535. We should add these to the Flume User Guide.

      1. FLUME-1621.patch
        2 kB
        Israel Ekpo
      2. FLUME-1621.2.patch
        2 kB
        Israel Ekpo

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in flume-trunk #444 (See https://builds.apache.org/job/flume-trunk/444/)
          FLUME-1621. Document new MemoryChannel parameters in Flume User Guide. (Revision 4c9e85ab5582f50e2faba9ab1f01d60395c5fc90)

          Result = FAILURE
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4c9e85ab5582f50e2faba9ab1f01d60395c5fc90
          Files :

          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          Show
          Hudson added a comment - Integrated in flume-trunk #444 (See https://builds.apache.org/job/flume-trunk/444/ ) FLUME-1621 . Document new MemoryChannel parameters in Flume User Guide. (Revision 4c9e85ab5582f50e2faba9ab1f01d60395c5fc90) Result = FAILURE mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4c9e85ab5582f50e2faba9ab1f01d60395c5fc90 Files : flume-ng-doc/sphinx/FlumeUserGuide.rst
          Hide
          Mike Percy added a comment -

          Pushed to trunk and flume-1.4 branches. Thanks Israel!

          Show
          Mike Percy added a comment - Pushed to trunk and flume-1.4 branches. Thanks Israel!
          Hide
          Mike Percy added a comment -

          +1

          Show
          Mike Percy added a comment - +1
          Hide
          Israel Ekpo added a comment -

          The patch for this is available here

          https://reviews.apache.org/r/10417/

          Please take a look at it so that we can close this out when you get a chance.

          Thanks.

          Show
          Israel Ekpo added a comment - The patch for this is available here https://reviews.apache.org/r/10417/ Please take a look at it so that we can close this out when you get a chance. Thanks.
          Hide
          Israel Ekpo added a comment -

          A review has been created for the updated patch

          https://reviews.apache.org/r/10417/

          Show
          Israel Ekpo added a comment - A review has been created for the updated patch https://reviews.apache.org/r/10417/
          Hide
          Israel Ekpo added a comment -

          Attaching newly updated patch

          Show
          Israel Ekpo added a comment - Attaching newly updated patch
          Hide
          Israel Ekpo added a comment - - edited

          Thanks Mike.

          It makes sense and will be less confusing to the end users.

          I will update the patch and re-attach the new version based on your recommendations.

          Show
          Israel Ekpo added a comment - - edited Thanks Mike. It makes sense and will be less confusing to the end users. I will update the patch and re-attach the new version based on your recommendations.
          Hide
          Mike Percy added a comment -

          Thanks for the patch Israel, looks pretty good but I have a couple suggestions:
          1. transactionCapacity is basically a safeguard, a sanity check, and any component that tries to put() or take() more than that number of events in a single transaction will get an exception thrown when they try. So we shouldn't characterize it as just a sink thing.
          2. In the example, making the capacity a power of 2 seems a bit confusing to me since it kind of seems like we're using 1KB of memory or something. A nice even number of events (like a power of ten) seems more natural to me. But setting it to a larger number such as 1000000 for memory channel seems realistic, that might be a better option. transactionCapacity would typically be set to a significantly smaller number than capacity. I'd say maybe 10000 is the largest batch size that is typically helpful.

          Show
          Mike Percy added a comment - Thanks for the patch Israel, looks pretty good but I have a couple suggestions: 1. transactionCapacity is basically a safeguard, a sanity check, and any component that tries to put() or take() more than that number of events in a single transaction will get an exception thrown when they try. So we shouldn't characterize it as just a sink thing. 2. In the example, making the capacity a power of 2 seems a bit confusing to me since it kind of seems like we're using 1KB of memory or something. A nice even number of events (like a power of ten) seems more natural to me. But setting it to a larger number such as 1000000 for memory channel seems realistic, that might be a better option. transactionCapacity would typically be set to a significantly smaller number than capacity. I'd say maybe 10000 is the largest batch size that is typically helpful.
          Hide
          Israel Ekpo added a comment - - edited

          I noticed that this was already updated in 1.4.0

          I will just add a few comments to what is already there.

          In the patch, I have also added some configuration settings that were absent for the sample agent using MemoryChannel

          Show
          Israel Ekpo added a comment - - edited I noticed that this was already updated in 1.4.0 I will just add a few comments to what is already there. In the patch, I have also added some configuration settings that were absent for the sample agent using MemoryChannel
          Hide
          Israel Ekpo added a comment -

          I will take this on and document the new parameters that could be configured for org.apache.flume.channel.MemoryChannel

          Show
          Israel Ekpo added a comment - I will take this on and document the new parameters that could be configured for org.apache.flume.channel.MemoryChannel

            People

            • Assignee:
              Israel Ekpo
              Reporter:
              Hari Shreedharan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Development