Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-281

Create pig scripts to perform down sampling

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.2.0
    • Component/s: Data Processors
    • Labels:
      None

      Description

      Use pig scripts instead of MySQL to perform down sampling.

        Activity

        Hide
        zhangyongjiang Cheng added a comment -

        Shell script downSampling.sh could be called from command line or cron job to generate the down sampling results.

        command line options:
        -c <space seperated cluster names>
        optional. if no cluster specified, hadoop dfs command will be used to get all clusters from /chukwa/repos
        -d <yyyyMMdd>
        optional. if no date is given, yesterday will be used
        -h help
        -m <space seperated metrics list>
        optional. Default value is 'SystemMetrics Hadoop_dfs_namenode Hadoop_dfs_FSDirectory Hadoop_dfs_datanode Hadoop_rpc_metrics Hadoop_mapred_jobtracker Hadoop_jvm_metrics Hadoop_dfs_FSNamesystem Df'
        -t time
        optional. default value is '5 30 180 720'
        -n <add|remove>
        Add/remove cron jobs.
        optional.
        start: add cron entry
        remove: remove cron entry

        Show
        zhangyongjiang Cheng added a comment - Shell script downSampling.sh could be called from command line or cron job to generate the down sampling results. command line options: -c <space seperated cluster names> optional. if no cluster specified, hadoop dfs command will be used to get all clusters from /chukwa/repos -d <yyyyMMdd> optional. if no date is given, yesterday will be used -h help -m <space seperated metrics list> optional. Default value is 'SystemMetrics Hadoop_dfs_namenode Hadoop_dfs_FSDirectory Hadoop_dfs_datanode Hadoop_rpc_metrics Hadoop_mapred_jobtracker Hadoop_jvm_metrics Hadoop_dfs_FSNamesystem Df' -t time optional. default value is '5 30 180 720' -n <add|remove> Add/remove cron jobs. optional. start: add cron entry remove: remove cron entry
        Hide
        eyang Eric Yang added a comment -

        This feature is fairly big. Please provide a test case for this feature. Thanks

        Show
        eyang Eric Yang added a comment - This feature is fairly big. Please provide a test case for this feature. Thanks
        Hide
        eyang Eric Yang added a comment -

        Pig scripts are not bundled as part of the tarball. Please update the build procedure.

        Show
        eyang Eric Yang added a comment - Pig scripts are not bundled as part of the tarball. Please update the build procedure.
        Hide
        zhangyongjiang Cheng added a comment -

        New patch created. Pig scripts are now included in the package. Will provide unit test case later.

        Show
        zhangyongjiang Cheng added a comment - New patch created. Pig scripts are now included in the package. Will provide unit test case later.
        Hide
        eyang Eric Yang added a comment -

        Looks like the patch contains non-related code like TaskLogAppender, and ChukwaJobInstrumentation.java. Please make sure only related code are in the patch.

        Show
        eyang Eric Yang added a comment - Looks like the patch contains non-related code like TaskLogAppender, and ChukwaJobInstrumentation.java. Please make sure only related code are in the patch.
        Hide
        zhangyongjiang Cheng added a comment -

        Regenerated the patch.

        Show
        zhangyongjiang Cheng added a comment - Regenerated the patch.
        Hide
        eyang Eric Yang added a comment -

        +1 looks good.

        Show
        eyang Eric Yang added a comment - +1 looks good.
        Hide
        eyang Eric Yang added a comment -

        I just committed this, thanks Cheng.

        Show
        eyang Eric Yang added a comment - I just committed this, thanks Cheng.
        Hide
        hudson Hudson added a comment -

        Integrated in Chukwa-trunk #49 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/49/)
        . Created pig scripts to perform down sampling. (Cheng Zhang via Eric Yang)

        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #49 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/49/ ) . Created pig scripts to perform down sampling. (Cheng Zhang via Eric Yang)

          People

          • Assignee:
            zhangyongjiang Cheng
            Reporter:
            zhangyongjiang Cheng
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development