Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-419

Replace system metrics collection with sigar

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: Data Collection
    • Labels:
      None
    • Environment:

      Redhat EL 5.3, Java 6

      Description

      Instead of having multiple java vm to execute unix commands to collect system metrics (CPU utilization, network, memory, disk), it's more efficient to collect the metrics through JNI interface. SIGAR is a good replacement for Chukwa's input tools. All licenses considered, it's the next best option than reinvent the wheel.

      1. CHUKWA-419-draft.patch
        13 kB
        Eric Yang
      2. CHUKWA-419-draft-2.patch
        14 kB
        Eric Yang

        Activity

        Hide
        asrabkin Ari Rabkin added a comment -

        Sigar is GPLed, so we can't redistribute it. I'm not sure we can even legally link against it. We probably COULD include a Sigar adaptor in contrib.

        Note that the exec adaptor is now working well, so running unix commands doesn't require launching JVMs.

        Show
        asrabkin Ari Rabkin added a comment - Sigar is GPLed, so we can't redistribute it. I'm not sure we can even legally link against it. We probably COULD include a Sigar adaptor in contrib. Note that the exec adaptor is now working well, so running unix commands doesn't require launching JVMs.
        Hide
        eyang Eric Yang added a comment -

        Won't fix due to license issues.

        Show
        eyang Eric Yang added a comment - Won't fix due to license issues.
        Hide
        eyang Eric Yang added a comment -

        Sigar has switched from GPL license to Apache License 2.0 for Sigar 1.6.4+. The website has not made the announcement, but SVN license has been updated for a while. Hence, it's worth while to revisit this issue.

        Show
        eyang Eric Yang added a comment - Sigar has switched from GPL license to Apache License 2.0 for Sigar 1.6.4+. The website has not made the announcement, but SVN license has been updated for a while. Hence, it's worth while to revisit this issue.
        Hide
        eyang Eric Yang added a comment -

        Usage:

        Run this in Agent:

        add org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics SystemMetrics 60 0

        This will generate system metrics stats every 60 seconds, and serialized in JSON format. A demux parser is also supplied in this patch for storing system metrics.

        Show
        eyang Eric Yang added a comment - Usage: Run this in Agent: add org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics SystemMetrics 60 0 This will generate system metrics stats every 60 seconds, and serialized in JSON format. A demux parser is also supplied in this patch for storing system metrics.
        Hide
        eyang Eric Yang added a comment -

        Updated license for demux parser.

        Show
        eyang Eric Yang added a comment - Updated license for demux parser.
        Hide
        asrabkin Ari Rabkin added a comment -

        Patch looks good. I actually would use this. Is it committable?

        Show
        asrabkin Ari Rabkin added a comment - Patch looks good. I actually would use this. Is it committable?
        Hide
        eyang Eric Yang added a comment -

        The patch works. I flagged it as draft due to lack of automated method to download sigar 1.6.5 from maven repository. I decided to bundle Sigar 1.6.5 with license.txt as part of our library source code instead of waiting for Sigar team to publish 1.6.5 binary to maven. I tested on MacOSX only, but it should work with all major platforms. I just committed this.

        Show
        eyang Eric Yang added a comment - The patch works. I flagged it as draft due to lack of automated method to download sigar 1.6.5 from maven repository. I decided to bundle Sigar 1.6.5 with license.txt as part of our library source code instead of waiting for Sigar team to publish 1.6.5 binary to maven. I tested on MacOSX only, but it should work with all major platforms. I just committed this.

          People

          • Assignee:
            eyang Eric Yang
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development