Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: Scripts
    • Labels:
      None
    • Environment:

      Java 6, Mac OS X 10.6

      Description

      Chukwa record metrics from name node, data node, job tracker, task tracker, etc, but the raw metrics does not help determine all aspect of the cluster health. For now, we have the following metrics in HBase:

      • System
      • Disk
      • Memory
      • Network
      • HDFS
      • Name Node
      • Data Node
      • Map Reduce
      • Job Tracker
      • Task Tracker

      We can further analyze the data to provide a summary for the cluster as these categories:

      • System - Performance profile of how busy the nodes are in the cluster
      • HDFS - Capacity of the disk storage, and health of the data in the file system
      • MapReduce - Capacity of the processing pipeline, and health of the processing system

        Activity

        Hide
        eyang Eric Yang added a comment -

        Usage:

        pig -param START=[starting epoch millisecond] -Dpig.additional.jars=$PIG_PATH/pig.jar:$HBASE_HOME/hbase-0.20.6.jar $CHUKWA_HOME/script/pig/ClusterSummary.pig

        Show
        eyang Eric Yang added a comment - Usage: pig -param START= [starting epoch millisecond] -Dpig.additional.jars=$PIG_PATH/pig.jar:$HBASE_HOME/hbase-0.20.6.jar $CHUKWA_HOME/script/pig/ClusterSummary.pig
        Hide
        eyang Eric Yang added a comment -

        This script aggregates:

        • cpu, disk, memory, network usage
        • hdfs space capacity, hdfs space remaining, errors status
        • mapreduce slot capacity, slot usage, error status

        by cluster, and store in ClusterSummary table.

        Show
        eyang Eric Yang added a comment - This script aggregates: cpu, disk, memory, network usage hdfs space capacity, hdfs space remaining, errors status mapreduce slot capacity, slot usage, error status by cluster, and store in ClusterSummary table.
        Hide
        asrabkin Ari Rabkin added a comment -

        Looks OK. +1 to commit. I will test on my cluster.

        Show
        asrabkin Ari Rabkin added a comment - Looks OK. +1 to commit. I will test on my cluster.
        Hide
        asrabkin Ari Rabkin added a comment -

        Tried this, got errors.

        I started with a clean HBase, let it collect metrics from the default adaptors for a bit. Ran the script manually. The Pig-spawned tasks all fail. I got the following on the Reduce side:

        java.io.IOException: java.lang.IllegalArgumentException: Row key is invalid
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:438)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:401)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:381)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:251)
        at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:566)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
        Caused by: java.lang.IllegalArgumentException: Row key is invalid
        at org.apache.hadoop.hbase.client.Put.(Put.java:79)
        at org.apache.hadoop.hbase.client.Put.(Put.java:69)
        at org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:355)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
        at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:508)
        at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:436)
        ... 7 more

        ------

        Is it possible to run the scripts in local mode for debugging, and have them still pull data from HBase? How do I configure that? I tried a bunch of things and got nowhere.

        Show
        asrabkin Ari Rabkin added a comment - Tried this, got errors. I started with a clean HBase, let it collect metrics from the default adaptors for a bit. Ran the script manually. The Pig-spawned tasks all fail. I got the following on the Reduce side: java.io.IOException: java.lang.IllegalArgumentException: Row key is invalid at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:438) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:401) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:381) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:251) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:566) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408) at org.apache.hadoop.mapred.Child.main(Child.java:170) Caused by: java.lang.IllegalArgumentException: Row key is invalid at org.apache.hadoop.hbase.client.Put.(Put.java:79) at org.apache.hadoop.hbase.client.Put.(Put.java:69) at org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:355) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:138) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97) at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:508) at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:436) ... 7 more ------ Is it possible to run the scripts in local mode for debugging, and have them still pull data from HBase? How do I configure that? I tried a bunch of things and got nowhere.
        Hide
        asrabkin Ari Rabkin added a comment -

        CHUKWA-578 blocks this.

        Show
        asrabkin Ari Rabkin added a comment - CHUKWA-578 blocks this.
        Hide
        eyang Eric Yang added a comment -

        My configuration for running hbase+pig:

        Single node hadoop+hbase+chukwa:

        export PIG_PATH=sandbox/pig-0.8.0
        export PIG_CLASSPATH=${HBASE_CONF_DIR}:${HADOOP_CONF_DIR}
        export HBASE_HOME=sandbox/hbase-0.20.6
        export CHUKWA_HOME=sandbox/chukwa-trunk
        
        ./pig -Dpig.additional.jars=${PIG_PATH}/pig-0.8.0-core.jar:${HBASE_HOME}/hbase-0.20.6.jar ${CHUKWA_HOME}/script/pig/ClusterSummary.pig
        

        Real cluster + Mapreduce:

        create a jar file containing hbase-site.xml, and call it hbase-conf.jar

        export PIG_PATH=sandbox/pig-0.8.0
        export PIG_CLASSPATH=${HBASE_CONF_DIR}:${HADOOP_CONF_DIR}
        export HBASE_HOME=sandbox/hbase-0.20.6
        export CHUKWA_HOME=sandbox/chukwa-trunk
        
        ./pig -Dpig.additional.jars=${PIG_PATH}/pig-0.8.0-core.jar:${HBASE_HOME}/hbase-0.20.6.jar:hbase-conf.jar ${CHUKWA_HOME}/script/pig/ClusterSummary.pig
        

        See if you can get this to work, try to run the script in grunt mode line by line, and inspect which STORE statement is causing problem.

        Show
        eyang Eric Yang added a comment - My configuration for running hbase+pig: Single node hadoop+hbase+chukwa: export PIG_PATH=sandbox/pig-0.8.0 export PIG_CLASSPATH=${HBASE_CONF_DIR}:${HADOOP_CONF_DIR} export HBASE_HOME=sandbox/hbase-0.20.6 export CHUKWA_HOME=sandbox/chukwa-trunk ./pig -Dpig.additional.jars=${PIG_PATH}/pig-0.8.0-core.jar:${HBASE_HOME}/hbase-0.20.6.jar ${CHUKWA_HOME}/script/pig/ClusterSummary.pig Real cluster + Mapreduce: create a jar file containing hbase-site.xml, and call it hbase-conf.jar export PIG_PATH=sandbox/pig-0.8.0 export PIG_CLASSPATH=${HBASE_CONF_DIR}:${HADOOP_CONF_DIR} export HBASE_HOME=sandbox/hbase-0.20.6 export CHUKWA_HOME=sandbox/chukwa-trunk ./pig -Dpig.additional.jars=${PIG_PATH}/pig-0.8.0-core.jar:${HBASE_HOME}/hbase-0.20.6.jar:hbase-conf.jar ${CHUKWA_HOME}/script/pig/ClusterSummary.pig See if you can get this to work, try to run the script in grunt mode line by line, and inspect which STORE statement is causing problem.
        Hide
        eyang Eric Yang added a comment -

        Since CHUKWA-578 is invalid, resubmit patch status again.

        Show
        eyang Eric Yang added a comment - Since CHUKWA-578 is invalid, resubmit patch status again.
        Hide
        eyang Eric Yang added a comment -

        I just committed this.

        Show
        eyang Eric Yang added a comment - I just committed this.

          People

          • Assignee:
            eyang Eric Yang
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development