Flume
  1. Flume
  2. FLUME-1411

Add average events per second params to MBeans

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.2.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      I know this might cause the platform mbean thread longer to poll, since we will need to do some calculation - but it is definitely nice to have historic averages.

        Issue Links

          Activity

          Hide
          Ted Malaska added a comment -

          Hey Hari,

          My question is "get the average per second over what time period"?

          One possible implementation would be to added the following methods to MonitoredCounterGroup:

          getAverage(String counter) //This will return the average of events since last AvgAsOfTime
          resetAverages() // This will reset AvgAsOfTime to now

          Note: AvgAsOfTime will start as the time the counter was created.

          This will need very little memory or processing power. It will also allow the user the ability to get Averages for what ever time interval length they want.

          Show
          Ted Malaska added a comment - Hey Hari, My question is "get the average per second over what time period"? One possible implementation would be to added the following methods to MonitoredCounterGroup: getAverage(String counter) //This will return the average of events since last AvgAsOfTime resetAverages() // This will reset AvgAsOfTime to now Note: AvgAsOfTime will start as the time the counter was created. This will need very little memory or processing power. It will also allow the user the ability to get Averages for what ever time interval length they want.
          Hide
          Hari Shreedharan added a comment -

          My intention was just to show the averages events/sec since the component started. So I didn't really have reset() in mind. What purpose does a reset() method serve? The mbean is not visible from outside the component, so when would you reset the average? I don't really see any need for a reset. Simply implementing getAverage would be enough.

          Note: Mbean methods are simply getters. They do not take an argument. So you would have a method called getAverageEventRate() or something, in the ChannelCounter interface, and its implementation in ChannelCounter class.

          An easy way to implement this is to have 2 methods, one for average puts per second, and one for average takes per second. Use the existing mbean data like putSuccessCount and startTime to calculate the average.

          Show
          Hari Shreedharan added a comment - My intention was just to show the averages events/sec since the component started. So I didn't really have reset() in mind. What purpose does a reset() method serve? The mbean is not visible from outside the component, so when would you reset the average? I don't really see any need for a reset. Simply implementing getAverage would be enough. Note: Mbean methods are simply getters. They do not take an argument. So you would have a method called getAverageEventRate() or something, in the ChannelCounter interface, and its implementation in ChannelCounter class. An easy way to implement this is to have 2 methods, one for average puts per second, and one for average takes per second. Use the existing mbean data like putSuccessCount and startTime to calculate the average.
          Hide
          Ted Malaska added a comment -

          I can do what you ask. But let me clarify and ask one more question.

          1. Clarify
          When I said getAverage(String counter) in MonitoredCounterGroup it is like

          • getAverage(COUNTER_EVENT_TAKE_ATTEMPT) is to getAverageEventRate() as get(COUNTER_EVENT_TAKE_ATTEMPT) getEventTakeAttemptCount.

          So on that note, we are saying the same thing.

          2. Question
          When you say "averages events/sec since the component started" it is a simple a dividing the number of events by the number of second since the component started.

          The problem with that approach is the longer the component runs the less reactive the result of getAverage will be to changes to events per second. So the value becomes less valuable to the user the longer the component runs.

          What would be really cool if the user could get like the average for the last N minutes and chart it in something like http://mbostock.github.com/d3/ex/stack.html

          The reset is one way to do this. Another way to do this is change the method to say something like getLastMinuteAverageOfFoo()

          Do you see what I saying?

          Show
          Ted Malaska added a comment - I can do what you ask. But let me clarify and ask one more question. 1. Clarify When I said getAverage(String counter) in MonitoredCounterGroup it is like getAverage(COUNTER_EVENT_TAKE_ATTEMPT) is to getAverageEventRate() as get(COUNTER_EVENT_TAKE_ATTEMPT) getEventTakeAttemptCount. So on that note, we are saying the same thing. 2. Question When you say "averages events/sec since the component started" it is a simple a dividing the number of events by the number of second since the component started. The problem with that approach is the longer the component runs the less reactive the result of getAverage will be to changes to events per second. So the value becomes less valuable to the user the longer the component runs. What would be really cool if the user could get like the average for the last N minutes and chart it in something like http://mbostock.github.com/d3/ex/stack.html The reset is one way to do this. Another way to do this is change the method to say something like getLastMinuteAverageOfFoo() Do you see what I saying?
          Hide
          Hari Shreedharan added a comment -

          I don't mind having something that shows the average the way you mention. The only issue with that is that you need to have a scheduled executor doing that every n seconds/minutes, right? Where would this go? Copy-paste in each of the components ?

          Show
          Hari Shreedharan added a comment - I don't mind having something that shows the average the way you mention. The only issue with that is that you need to have a scheduled executor doing that every n seconds/minutes, right? Where would this go? Copy-paste in each of the components ?
          Hide
          Ted Malaska added a comment -

          I don't this so. I have an idea.

          I'll try to get a patch made without unit test to show my idea.

          If you like it, I'll make unit tests.

          Show
          Ted Malaska added a comment - I don't this so. I have an idea. I'll try to get a patch made without unit test to show my idea. If you like it, I'll make unit tests.
          Hide
          Denny Ye added a comment -

          It's effective improvement for monitoring. If we can support the average throughput, that be more better https://issues.apache.org/jira/browse/FLUME-1484

          Show
          Denny Ye added a comment - It's effective improvement for monitoring. If we can support the average throughput, that be more better https://issues.apache.org/jira/browse/FLUME-1484
          Hide
          Ted Malaska added a comment -

          Yes. Lets link these two JIRAs together. I'll build something as soon as I finish my current task.

          Show
          Ted Malaska added a comment - Yes. Lets link these two JIRAs together. I'll build something as soon as I finish my current task.
          Hide
          Ted Malaska added a comment - - edited

          This is not a completely finished patch but an example of a possible solution. If you like the solution I'll finish the patch.

          This solution supports two types of averages:
          1. Averages since start
          2. Rolling Averages (The rolling interval can be set through config properties)

          I added two gets for these averages in sinkCounter.

          I also added a junit that tests both of these averages and shows how quickly they deviate.

          Please let me know what you think.

          Show
          Ted Malaska added a comment - - edited This is not a completely finished patch but an example of a possible solution. If you like the solution I'll finish the patch. This solution supports two types of averages: 1. Averages since start 2. Rolling Averages (The rolling interval can be set through config properties) I added two gets for these averages in sinkCounter. I also added a junit that tests both of these averages and shows how quickly they deviate. Please let me know what you think.
          Hide
          Ted Malaska added a comment -

          Just wondering if anyone had any feed back on 1411.

          Show
          Ted Malaska added a comment - Just wondering if anyone had any feed back on 1411.
          Hide
          Ted Malaska added a comment -

          Just wondering if anyone has any feed back. I would like to close this bug

          Show
          Ted Malaska added a comment - Just wondering if anyone has any feed back. I would like to close this bug

            People

            • Assignee:
              Ted Malaska
              Reporter:
              Hari Shreedharan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development