UIMA
  1. UIMA
  2. UIMA-1104

Need a monitor component for UIMA-AS services to capture performance metrics

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3AS
    • Component/s: Async Scaleout
    • Labels:
      None

      Description

      In complex uima-as deployments it is hard to find bottlenecks which need scaleup. A JMX-based monitor is needed to collect runtime metrics from every uima-as service. The metrics must include idle time, queue depth, amount of time each service waits for a free CAS. The monitor should be an embeddable component that can be deployed in a java application. The monitor should allow custom formatting of metrics via pluggable extension.

      1. idleWithRemote.txt
        23 kB
        Eddie Epstein
      2. uimaj-as-activemq-UIMA-1104-patch.txt
        17 kB
        Jerry Cwiklik
      3. uimaj-as-activemq-UIMA-1104-patch-03.txt
        0.8 kB
        Jerry Cwiklik
      4. uimaj-as-activemq-UIMA-1104-patch-04.txt
        4 kB
        Jerry Cwiklik
      5. uimaj-as-activemq-UIMA-1104-patch-05.txt
        3 kB
        Jerry Cwiklik
      6. uimaj-as-activemq-UIMA-1104-patch-06.txt
        3 kB
        Jerry Cwiklik
      7. uimaj-as-activemq-UIMA-1104-patch-07.txt
        34 kB
        Jerry Cwiklik
      8. uimaj-as-activemq-UIMA-1104-patch-09.txt
        15 kB
        Jerry Cwiklik
      9. uimaj-as-activemq-UIMA-1104-patch-10.txt
        5 kB
        Jerry Cwiklik
      10. uimaj-as-activemq-UIMA-1104-patch-11.txt
        1 kB
        Jerry Cwiklik
      11. uimaj-as-activemq-UIMA-1104-patch-12.txt
        3 kB
        Jerry Cwiklik
      12. uimaj-as-activemq-UIMA-1104-patch-13.txt
        11 kB
        Jerry Cwiklik
      13. uimaj-as-core-UIMA-1104-patch.txt
        11 kB
        Jerry Cwiklik
      14. uimaj-as-core-UIMA-1104-patch-02.txt
        13 kB
        Jerry Cwiklik
      15. uimaj-as-core-UIMA-1104-patch-03.txt
        2 kB
        Jerry Cwiklik
      16. uimaj-as-core-UIMA-1104-patch-04.txt
        9 kB
        Jerry Cwiklik
      17. uimaj-as-core-UIMA-1104-patch-05.txt
        9 kB
        Jerry Cwiklik
      18. uimaj-as-core-UIMA-1104-patch-06.txt
        6 kB
        Jerry Cwiklik
      19. uimaj-as-core-UIMA-1104-patch-08.txt
        12 kB
        Jerry Cwiklik
      20. uimaj-as-core-UIMA-1104-patch-09.txt
        28 kB
        Jerry Cwiklik
      21. uimaj-as-core-UIMA-1104-patch-10.txt
        12 kB
        Jerry Cwiklik
      22. uimaj-as-core-UIMA-1104-patch-12.txt
        15 kB
        Jerry Cwiklik
      23. uimaj-as-core-UIMA-1104-patch-13.txt
        11 kB
        Jerry Cwiklik

        Activity

        Hide
        Jerry Cwiklik added a comment -

        Created a new uima-as component to capture runtime metrics of uima-as services. The monitor connects to a JMX server via provided URI and begins checkpoints at a given intervals each time collecting metrics from JMX service beans. The monitor supports plug in listener that are notified each time the checkpoint is done. The listener receives the metrics for the current checkpoint (interval). The listener can format the metrics to specific requirements.

        Show
        Jerry Cwiklik added a comment - Created a new uima-as component to capture runtime metrics of uima-as services. The monitor connects to a JMX server via provided URI and begins checkpoints at a given intervals each time collecting metrics from JMX service beans. The monitor supports plug in listener that are notified each time the checkpoint is done. The listener receives the metrics for the current checkpoint (interval). The listener can format the metrics to specific requirements.
        Hide
        Jerry Cwiklik added a comment -

        Fixed how CASes are counted. The service should only count CASes coming from its input queue.

        Show
        Jerry Cwiklik added a comment - Fixed how CASes are counted. The service should only count CASes coming from its input queue.
        Hide
        Eddie Epstein added a comment -

        Jerry,

        Something wrong with the stats below. Scenario was:

        service = Deploy_MeetingDetectorTAE_Sync_3Instances.xml
        client = runremoteasyncae tcp://localhost:61616 MeetingDetectorTaeQueue -c ManyDocsCollectionReader.xml -p 15 -f 100000

        ManyDocsCollectionReader.xml is pointing a directory with a few thousand documents with sizes ranging from 1000-10000 chars.

        INFO: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
        Jul 8, 2008 9:51:31 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: Timestamp: 6,855 [ TutorialDateTime Annotator_Service Performance ] Idle 1,021 CASes 0 InQDepth 0 SCP Wait 0 CMgetNext 0
        Jul 8, 2008 9:51:31 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: Timestamp: 6,855 [ Meeting Detector TAE_Service Performance ] Idle 1,022 CASes 0 InQDepth 0 SCP Wait 0 CMgetNext 0
        Jul 8, 2008 9:51:31 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: Timestamp: 6,855 [ Room Number Annotator_Service Performance ] Idle 1,020 CASes 0 InQDepth 0 SCP Wait 0 CMgetNext 0
        Jul 8, 2008 9:51:31 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: Timestamp: 6,855 [ Meeting Annotator_Service Performance ] Idle 1,020 CASes 0 InQDepth 0 SCP Wait 0 CMgetNext 0
        Jul 8, 2008 9:51:31 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
        Jul 8, 2008 9:51:32 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: Timestamp: 7,889 [ TutorialDateTime Annotator_Service Performance ] Idle 1,032 CASes 7 InQDepth 0 SCP Wait 0 CMgetNext 0
        Jul 8, 2008 9:51:32 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: Timestamp: 7,889 [ Meeting Detector TAE_Service Performance ] Idle 1,140 CASes 7 InQDepth 0 SCP Wait 24 CMgetNext 0
        Jul 8, 2008 9:51:32 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: Timestamp: 7,889 [ Room Number Annotator_Service Performance ] Idle 1,032 CASes 7 InQDepth 0 SCP Wait 0 CMgetNext 0
        Jul 8, 2008 9:51:32 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: Timestamp: 7,889 [ Meeting Annotator_Service Performance ] Idle 1,046 CASes 7 InQDepth 0 SCP Wait 0 CMgetNext 0
        Jul 8, 2008 9:51:32 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
        Jul 8, 2008 9:51:33 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: Timestamp: 8,948 [ TutorialDateTime Annotator_Service Performance ] Idle 1,043 CASes 62 InQDepth 0 SCP Wait 0 CMgetNext 0
        Jul 8, 2008 9:51:33 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: Timestamp: 8,948 [ Meeting Detector TAE_Service Performance ] Idle 41,553 CASes 65 InQDepth 9 SCP Wait 427 CMgetNext 0
        Jul 8, 2008 9:51:33 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: Timestamp: 8,948 [ Room Number Annotator_Service Performance ] Idle 884 CASes 64 InQDepth 0 SCP Wait 0 CMgetNext 0
        Jul 8, 2008 9:51:33 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        INFO: Timestamp: 8,948 [ Meeting Annotator_Service Performance ] Idle 1,051 CASes 61 InQDepth 0 SCP Wait 0 CMgetNext 0
        Jul 8, 2008 9:51:33 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run

        Show
        Eddie Epstein added a comment - Jerry, Something wrong with the stats below. Scenario was: service = Deploy_MeetingDetectorTAE_Sync_3Instances.xml client = runremoteasyncae tcp://localhost:61616 MeetingDetectorTaeQueue -c ManyDocsCollectionReader.xml -p 15 -f 100000 ManyDocsCollectionReader.xml is pointing a directory with a few thousand documents with sizes ranging from 1000-10000 chars. INFO: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Jul 8, 2008 9:51:31 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: Timestamp: 6,855 [ TutorialDateTime Annotator_Service Performance ] Idle 1,021 CASes 0 InQDepth 0 SCP Wait 0 CMgetNext 0 Jul 8, 2008 9:51:31 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: Timestamp: 6,855 [ Meeting Detector TAE_Service Performance ] Idle 1,022 CASes 0 InQDepth 0 SCP Wait 0 CMgetNext 0 Jul 8, 2008 9:51:31 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: Timestamp: 6,855 [ Room Number Annotator_Service Performance ] Idle 1,020 CASes 0 InQDepth 0 SCP Wait 0 CMgetNext 0 Jul 8, 2008 9:51:31 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: Timestamp: 6,855 [ Meeting Annotator_Service Performance ] Idle 1,020 CASes 0 InQDepth 0 SCP Wait 0 CMgetNext 0 Jul 8, 2008 9:51:31 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Jul 8, 2008 9:51:32 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: Timestamp: 7,889 [ TutorialDateTime Annotator_Service Performance ] Idle 1,032 CASes 7 InQDepth 0 SCP Wait 0 CMgetNext 0 Jul 8, 2008 9:51:32 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: Timestamp: 7,889 [ Meeting Detector TAE_Service Performance ] Idle 1,140 CASes 7 InQDepth 0 SCP Wait 24 CMgetNext 0 Jul 8, 2008 9:51:32 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: Timestamp: 7,889 [ Room Number Annotator_Service Performance ] Idle 1,032 CASes 7 InQDepth 0 SCP Wait 0 CMgetNext 0 Jul 8, 2008 9:51:32 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: Timestamp: 7,889 [ Meeting Annotator_Service Performance ] Idle 1,046 CASes 7 InQDepth 0 SCP Wait 0 CMgetNext 0 Jul 8, 2008 9:51:32 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Jul 8, 2008 9:51:33 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: Timestamp: 8,948 [ TutorialDateTime Annotator_Service Performance ] Idle 1,043 CASes 62 InQDepth 0 SCP Wait 0 CMgetNext 0 Jul 8, 2008 9:51:33 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: Timestamp: 8,948 [ Meeting Detector TAE_Service Performance ] Idle 41,553 CASes 65 InQDepth 9 SCP Wait 427 CMgetNext 0 Jul 8, 2008 9:51:33 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: Timestamp: 8,948 [ Room Number Annotator_Service Performance ] Idle 884 CASes 64 InQDepth 0 SCP Wait 0 CMgetNext 0 Jul 8, 2008 9:51:33 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run INFO: Timestamp: 8,948 [ Meeting Annotator_Service Performance ] Idle 1,051 CASes 61 InQDepth 0 SCP Wait 0 CMgetNext 0 Jul 8, 2008 9:51:33 AM org.apache.uima.aae.jmx.monitor.JmxMonitorListener run
        Hide
        Jerry Cwiklik added a comment -

        Modified UIMA-AS service to call endProcess() if the service is not a Cas Multiplier. Fixes invalid idleTime.

        Show
        Jerry Cwiklik added a comment - Modified UIMA-AS service to call endProcess() if the service is not a Cas Multiplier. Fixes invalid idleTime.
        Hide
        Eddie Epstein added a comment -

        Looking much better. The attached log has a remote delegate. Not sure about how CASes are trickling out at the end.

        Show
        Eddie Epstein added a comment - Looking much better. The attached log has a remote delegate. Not sure about how CASes are trickling out at the end.
        Hide
        Jerry Cwiklik added a comment -

        Fixed idle time for remotes

        Show
        Jerry Cwiklik added a comment - Fixed idle time for remotes
        Hide
        Jerry Cwiklik added a comment -

        Removed debugging output from the code

        Show
        Jerry Cwiklik added a comment - Removed debugging output from the code
        Hide
        Jerry Cwiklik added a comment -

        Integrated JmxMonitor with the UIMA_Service. Refactored UIMA_Service to remove dead code.
        The JmxMonitor startup is enabled via -Djmx.monitor.frequency=<number> parameter. A presence of this parameter enables the monitor and redefines monitor's default checkpoint frequency. Also, the monitor supports pluggable formatter listeners. These listeners are called with new metrics every time a checkpoint is completed. The listener may format the metrics to specific requirements. To plug in a custom formatter use -Djmx.monitor.formatter=<class name> parameter.

        An example command line setting for the JmxMonitor is as follows:
        set UIMA_JVM_OPTS=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8009 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djmx.monitor.frequency=2000 -Djmx.monitor.formatter=org.apache.uima.examples.as.DefaultUimaServiceMonitorFormatter

        Note: the custom formatter class is instantiated using default (empty) constructor.

        Show
        Jerry Cwiklik added a comment - Integrated JmxMonitor with the UIMA_Service. Refactored UIMA_Service to remove dead code. The JmxMonitor startup is enabled via -Djmx.monitor.frequency=<number> parameter. A presence of this parameter enables the monitor and redefines monitor's default checkpoint frequency. Also, the monitor supports pluggable formatter listeners. These listeners are called with new metrics every time a checkpoint is completed. The listener may format the metrics to specific requirements. To plug in a custom formatter use -Djmx.monitor.formatter=<class name> parameter. An example command line setting for the JmxMonitor is as follows: set UIMA_JVM_OPTS=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8009 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djmx.monitor.frequency=2000 -Djmx.monitor.formatter=org.apache.uima.examples.as.DefaultUimaServiceMonitorFormatter Note: the custom formatter class is instantiated using default (empty) constructor.
        Hide
        Jerry Cwiklik added a comment -

        Changes to enable integration of JmxMonitor with the UIMA AS service wrapper

        Show
        Jerry Cwiklik added a comment - Changes to enable integration of JmxMonitor with the UIMA AS service wrapper
        Hide
        Jerry Cwiklik added a comment -

        Fixed the code that aggregates time spent waiting for CASes on a Cas Pool and a Shadow Cas Pool.

        Show
        Jerry Cwiklik added a comment - Fixed the code that aggregates time spent waiting for CASes on a Cas Pool and a Shadow Cas Pool.
        Hide
        Jerry Cwiklik added a comment -

        Modified to use more accurate thread timer. The thread CPU utilization is retrieved from the JVM MXBean. In case the JVM MXBean does not provide per thread timing info, a wall clock is still used. Removed time waiting in GetNext from the metrics, since it does not provide an accurate measure. Since the thread is sleeping while waiting (in getNext()) for a free CAS, the CPU utilization will be 0. Added two new attributes to the service metrics: threadCount which indicates number of process threads, and the analysis time which is time spent analyzing a CAS (for primitives). The Aggregate sums up all analysis times reported by its collocated delegates.

        Show
        Jerry Cwiklik added a comment - Modified to use more accurate thread timer. The thread CPU utilization is retrieved from the JVM MXBean. In case the JVM MXBean does not provide per thread timing info, a wall clock is still used. Removed time waiting in GetNext from the metrics, since it does not provide an accurate measure. Since the thread is sleeping while waiting (in getNext()) for a free CAS, the CPU utilization will be 0. Added two new attributes to the service metrics: threadCount which indicates number of process threads, and the analysis time which is time spent analyzing a CAS (for primitives). The Aggregate sums up all analysis times reported by its collocated delegates.
        Hide
        Jerry Cwiklik added a comment -

        Enhanced JMX Monitor to obtain free CAS instance count from Cas Multiplier's Cas Pool MBean.

        Show
        Jerry Cwiklik added a comment - Enhanced JMX Monitor to obtain free CAS instance count from Cas Multiplier's Cas Pool MBean.
        Hide
        Jerry Cwiklik added a comment -

        Fixed sun specific import if JMX OperatingSystemMXBean. Replaced it with a generic class from java.lang.management package

        Show
        Jerry Cwiklik added a comment - Fixed sun specific import if JMX OperatingSystemMXBean. Replaced it with a generic class from java.lang.management package
        Hide
        Jerry Cwiklik added a comment -

        Modified how the analysis time is aggregated and reported for JMX. Each time the JMX MBean asks for a current analysis time, the controller sums up the time of each processing thread and subtracts the time it takes to serialize and deserialize CASes. For aggregates, the analysis time is a sum of analysis times from all colocated delegates.

        Show
        Jerry Cwiklik added a comment - Modified how the analysis time is aggregated and reported for JMX. Each time the JMX MBean asks for a current analysis time, the controller sums up the time of each processing thread and subtracts the time it takes to serialize and deserialize CASes. For aggregates, the analysis time is a sum of analysis times from all colocated delegates.
        Hide
        Jerry Cwiklik added a comment -

        Added support for reporting the depth of reply queues. Collocated services reply to one (common) internal reply queue that is associated with an aggregate service. The collocated service reply queue depth will be reported as -1. Changed the API for retrieving the input queue depth.

        Show
        Jerry Cwiklik added a comment - Added support for reporting the depth of reply queues. Collocated services reply to one (common) internal reply queue that is associated with an aggregate service. The collocated service reply queue depth will be reported as -1. Changed the API for retrieving the input queue depth.
        Hide
        Eddie Epstein added a comment -

        Finally looks good. Any further changes should have specific issues created.

        Show
        Eddie Epstein added a comment - Finally looks good. Any further changes should have specific issues created.

          People

          • Assignee:
            Unassigned
            Reporter:
            Jerry Cwiklik
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development