Flume
  1. Flume
  2. FLUME-1940

Log a snapshot of Flume metrics on shutdown

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.3.1
    • Fix Version/s: v1.4.0
    • Component/s: None
    • Labels:

      Description

      It would be great to log a JMX snapshot to the flume.log at Flume shutdown time.

      1. FLUME-1940.2.patch
        4 kB
        Israel Ekpo
      2. FLUME-1940.20130424.patch
        7 kB
        Israel Ekpo
      3. FLUME-1940.3.patch
        6 kB
        Israel Ekpo
      4. FLUME-1940.patch
        3 kB
        Israel Ekpo

        Issue Links

          Activity

          Hide
          Israel Ekpo added a comment -

          I have created a patch for this.

          Will be attaching the patch shortly.

          Show
          Israel Ekpo added a comment - I have created a patch for this. Will be attaching the patch shortly.
          Hide
          Mike Percy added a comment -

          Israel, very nice! This is a great start.

          Some suggestions:
          1. It would be great to add this generic logging functionality to MonitoredCounterGroup
          2. Do the same as you did with ChannelCounter to SourceCounter and SinkCounter and just call the above impl from stop()
          3. It would be great to sort the metric key names before printing. As an example:

          List<String> keys = new ArrayList<String>(getCounterMap().keySet());
          Collections.sort(keys);
          for (String key : keys) { ... }
          

          4. Nit: minor indentation issue in the new MonitoredCounterGroup.getName() method (but you may not need to expose that if the impl is there)
          5. It would be great to also post this patch on https://reviews.apache.org/ per https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute#HowtoContribute-ProvidingPatches - use group Flume - and cross-link them. See other reviews at https://reviews.apache.org/groups/Flume/ as an example

          Thanks!

          Show
          Mike Percy added a comment - Israel, very nice! This is a great start. Some suggestions: 1. It would be great to add this generic logging functionality to MonitoredCounterGroup 2. Do the same as you did with ChannelCounter to SourceCounter and SinkCounter and just call the above impl from stop() 3. It would be great to sort the metric key names before printing. As an example: List<String> keys = new ArrayList<String>(getCounterMap().keySet()); Collections.sort(keys); for (String key : keys) { ... } 4. Nit: minor indentation issue in the new MonitoredCounterGroup.getName() method (but you may not need to expose that if the impl is there) 5. It would be great to also post this patch on https://reviews.apache.org/ per https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute#HowtoContribute-ProvidingPatches - use group Flume - and cross-link them. See other reviews at https://reviews.apache.org/groups/Flume/ as an example Thanks!
          Hide
          Mike Percy added a comment -

          Hi Israel, any update on this one?

          Thanks,
          Mike

          Show
          Mike Percy added a comment - Hi Israel, any update on this one? Thanks, Mike
          Hide
          Israel Ekpo added a comment -

          I started working on them last weekend.

          I should have a patch ready for review within the next 3 days.

          I think this issue will supersede FLUME-1957 when it is resolved and closed.

          Show
          Israel Ekpo added a comment - I started working on them last weekend. I should have a patch ready for review within the next 3 days. I think this issue will supersede FLUME-1957 when it is resolved and closed.
          Hide
          Mike Percy added a comment -

          Sounds great!

          Show
          Mike Percy added a comment - Sounds great!
          Hide
          Israel Ekpo added a comment -

          I plan to finish this either tonight or tomorrow night.

          I will put up the patch for review.

          Show
          Israel Ekpo added a comment - I plan to finish this either tonight or tomorrow night. I will put up the patch for review.
          Hide
          Israel Ekpo added a comment -

          This is the updated patch.

          Show
          Israel Ekpo added a comment - This is the updated patch.
          Hide
          Israel Ekpo added a comment -

          The review is available here

          https://reviews.apache.org/r/10416/

          Show
          Israel Ekpo added a comment - The review is available here https://reviews.apache.org/r/10416/
          Hide
          Mike Percy added a comment -

          Temporarily cancelling patch available due to RB feedback

          Show
          Mike Percy added a comment - Temporarily cancelling patch available due to RB feedback
          Hide
          Israel Ekpo added a comment -

          The updated patch has been submitted for review

          Show
          Israel Ekpo added a comment - The updated patch has been submitted for review
          Hide
          Mike Percy added a comment -

          +1

          Show
          Mike Percy added a comment - +1
          Hide
          Mike Percy added a comment -

          Thanks for the patch Israel! Pushed to trunk and flume-1.4 branches.

          Show
          Mike Percy added a comment - Thanks for the patch Israel! Pushed to trunk and flume-1.4 branches.
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #399 (See https://builds.apache.org/job/flume-trunk/399/)
          FLUME-1940. Log a snapshot of Flume metrics on shutdown. (Revision 215d75eb15362f34cd3246107cbea7ace247af10)

          Result = UNSTABLE
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=215d75eb15362f34cd3246107cbea7ace247af10
          Files :

          • flume-ng-core/src/main/java/org/apache/flume/instrumentation/MonitoredCounterGroup.java
          Show
          Hudson added a comment - Integrated in flume-trunk #399 (See https://builds.apache.org/job/flume-trunk/399/ ) FLUME-1940 . Log a snapshot of Flume metrics on shutdown. (Revision 215d75eb15362f34cd3246107cbea7ace247af10) Result = UNSTABLE mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=215d75eb15362f34cd3246107cbea7ace247af10 Files : flume-ng-core/src/main/java/org/apache/flume/instrumentation/MonitoredCounterGroup.java

            People

            • Assignee:
              Israel Ekpo
              Reporter:
              Mike Percy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Development