Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-571

Add digested system metrics to SystemMetrics demux parser

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: MR Data Processors
    • Labels:
      None
    • Environment:

      Java 6, Mac OS X 10.6

      Description

      For now, the system metrics demux parser are storing raw metrics. Each system components are measured independently. It would be nice to make a summary grouping for cpu, disk, network. Instead of storing cpu:combined.0 ... cpu:combined.8, for 8 cpu cores in the system, it would be nice to have a average number for all cpus combined. The digested metrics are easier to aggregate at cluster level.

        Activity

        Hide
        eyang Eric Yang added a comment -

        Digest metrics is parse and stored without number suffix. For example, the total bytes received by all network card is in network:RxBytes column. Network Loopback device is omitted from the summary number.

        Full list of digest metrics:

        • Averaged CPU utilization:
        • cpu:combined (user+sys)
        • cpu:idle
        • cpu:sys
        • cpu:user

        Sum of network devices throughput stats at interval:

        • network:RxBytes
        • network:RxDropped
        • network:RxErrors
        • network:RxPackets
        • network:TxBytes
        • network:TxColisions
        • network:TxErrors
        • network:TxPackets

        Sum of disk devices throughput stats at interval:

        • disk:ReadBytes
        • disk:Reads
        • disk:WriteBytes
        • disk:Writes
        Show
        eyang Eric Yang added a comment - Digest metrics is parse and stored without number suffix. For example, the total bytes received by all network card is in network:RxBytes column. Network Loopback device is omitted from the summary number. Full list of digest metrics: Averaged CPU utilization: cpu:combined (user+sys) cpu:idle cpu:sys cpu:user Sum of network devices throughput stats at interval: network:RxBytes network:RxDropped network:RxErrors network:RxPackets network:TxBytes network:TxColisions network:TxErrors network:TxPackets Sum of disk devices throughput stats at interval: disk:ReadBytes disk:Reads disk:WriteBytes disk:Writes
        Hide
        eyang Eric Yang added a comment -

        Patch for CHUKWA-572 should be applied with this patch for the metrics to generate properly.

        Show
        eyang Eric Yang added a comment - Patch for CHUKWA-572 should be applied with this patch for the metrics to generate properly.
        Hide
        asrabkin Ari Rabkin added a comment -

        Looks good; +1 to commit.

        Show
        asrabkin Ari Rabkin added a comment - Looks good; +1 to commit.
        Hide
        eyang Eric Yang added a comment -

        Thanks Ari. I just committed this.

        Show
        eyang Eric Yang added a comment - Thanks Ari. I just committed this.

          People

          • Assignee:
            eyang Eric Yang
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development