Chukwa
  1. Chukwa
  2. CHUKWA-306

Standalone (non-daemon) Chukwa operation

    Details

    • Type: Wish Wish
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      This is an articulation of a possible alternative use of Chukwa as a standalone log analysis pipeline. This would enable users to read in existing logs from files, process (Demux) and perform analysis (e.g. current SALSA/Mochi toolchain) on them, and visualize them, without requiring the user to setup or run any daemons, nor database servers.

      This can be presented as an alternative interface to Chukwa for the user, where the main architectural parts (Chunks, post-Demux SequenceFiles of ChukwaRecords, post-Demux-processing SequenceFiles of ChukwaRecords, and finally time-aggregated database entries for fast visualization) remain unchanged, and Chukwa is manifest as a set of files in HDFS. The main value that Chukwa then provides to users is 1. centralized one-stop-shop for log processing+analysis+anomaly detection, 2. the ability to use MapReduce to process logs, regardless of whether they had used Chukwa to collect the logs.

      That way, the ability to process logs and analyze/do diagnosis is not tied to having to run the entire Chukwa daemon infrastructure, since many users who use Hadoop clusters may not have superuser access to those machines, e.g. users at universities using shared clusters.

        Issue Links

          Activity

          Hide
          Jiaqi Tan added a comment -

          I would think there is some utility code to be written, and some config files to make Chukwa run without daemons, but this is mostly documentation. The backfilling tool already goes some way to meeting this, and the next thing we need to do is to remove the MySQL dependency but there's another JIRA for that.

          I am planning to submit a bunch of configs + scripts as a contrib module that would enable Chukwa to run without a Hadoop instance backing it using LocalJobRunner's and local storage.

          Show
          Jiaqi Tan added a comment - I would think there is some utility code to be written, and some config files to make Chukwa run without daemons, but this is mostly documentation. The backfilling tool already goes some way to meeting this, and the next thing we need to do is to remove the MySQL dependency but there's another JIRA for that. I am planning to submit a bunch of configs + scripts as a contrib module that would enable Chukwa to run without a Hadoop instance backing it using LocalJobRunner's and local storage.
          Hide
          Ari Rabkin added a comment -

          What's the to-do item here? Just documentation, or is there code to be written?

          Show
          Ari Rabkin added a comment - What's the to-do item here? Just documentation, or is there code to be written?
          Hide
          Jerome Boulon added a comment -

          Backfill do not open any port so there's no issue.
          But the port number cannot be randomized since you want to get jobHistory from Hadoop so Hadoop needs to know where to send the information.
          An option if you don't want to have one agent per machine, will be to use Zookeeper to coordinate everything but it will be more complicated to deploy not less

          Show
          Jerome Boulon added a comment - Backfill do not open any port so there's no issue. But the port number cannot be randomized since you want to get jobHistory from Hadoop so Hadoop needs to know where to send the information. An option if you don't want to have one agent per machine, will be to use Zookeeper to coordinate everything but it will be more complicated to deploy not less
          Hide
          Jiaqi Tan added a comment -

          Sounds good, but will there be port issues if we have multiple users trying to run Chukwa on the same host? e.g. on a shared gateway machine of a shared Hadoop cluster. My main target audience for a first "public trial" will be having M45 users (CMU or otherwise, since M45 is being opened up) be able to run this, and there are only a handful of gateway machines. Can the ant process also randomize ports and automagically fill in the config files with the port numbers?

          Show
          Jiaqi Tan added a comment - Sounds good, but will there be port issues if we have multiple users trying to run Chukwa on the same host? e.g. on a shared gateway machine of a shared Hadoop cluster. My main target audience for a first "public trial" will be having M45 users (CMU or otherwise, since M45 is being opened up) be able to run this, and there are only a handful of gateway machines. Can the ant process also randomize ports and automagically fill in the config files with the port numbers?
          Hide
          Jerome Boulon added a comment -

          By providing a system that is fully functional after running ant, i.e.:
          From CHUKWA_HOME:

          • only thing required: download, untar, run "ant"
          • default config should just work
          • bin/*.sh should just run
          • derbyDB alredy preloaded with some data
          • tomcat already preconfigured and ready to serve data from Derby
          • an example directory where we have an "HelloWorld" exampe for Chukwa
            --> shell script that uses the backfilling tools
            --> run demux locally
            --> load data to DerbyDb
            --> open the browser at the correct URL

          That will be a good start ....

          Show
          Jerome Boulon added a comment - By providing a system that is fully functional after running ant, i.e.: From CHUKWA_HOME: only thing required: download, untar, run "ant" default config should just work bin/*.sh should just run derbyDB alredy preloaded with some data tomcat already preconfigured and ready to serve data from Derby an example directory where we have an "HelloWorld" exampe for Chukwa --> shell script that uses the backfilling tools --> run demux locally --> load data to DerbyDb --> open the browser at the correct URL That will be a good start ....
          Hide
          Jiaqi Tan added a comment -

          Right, the backfilling tool is definitely part of the solution. The second part is how we can generate the visualizations without requiring the user to separately setup and configure Tomcat or any other web server, nor to setup and configure a separate database server.

          Show
          Jiaqi Tan added a comment - Right, the backfilling tool is definitely part of the solution. The second part is how we can generate the visualizations without requiring the user to separately setup and configure Tomcat or any other web server, nor to setup and configure a separate database server.
          Hide
          Jerome Boulon added a comment -

          The solution to this is to use the backfilling tool.
          This tool will produce your dataSink files (on per input file) then you just have to run demux like a standard M/R job.
          By configuring chukwa-collector-conf.xml to use the local filesystem then you don't have to worry about hadoop config at all.

          Show
          Jerome Boulon added a comment - The solution to this is to use the backfilling tool. This tool will produce your dataSink files (on per input file) then you just have to run demux like a standard M/R job. By configuring chukwa-collector-conf.xml to use the local filesystem then you don't have to worry about hadoop config at all.
          Hide
          Jiaqi Tan added a comment -

          You can get away without root access, but say you were on a shared environment, e.g. you are taking a class on Hadoop at school and you are using a shared server, and everyone logs onto a (handful of, at most) gateway machine to submit jobs to a shared cluster, and you wanted to crunch through your logs using Chukwa, you wouldn't want to sit around waiting 5 minutes for the collector to commit the chunks, or wait for the Demux to get run periodically, if you just wanted to look at the logs of one job. Also, more importantly, you wouldn't be able to run agents+adaptors on all of the slave nodes, and you would need to set up MySQL (wouldn't be easy if you're a not-very-systemsy, average user), and do a lot of additional setup steps that the average Hadoop user wouldn't want to invest the effort just to look at the analysis of logs from one job.

          The aim is to lower the amount of effort needed to use Chukwa for log/metric analysis and anomaly detection, visualization, etc.

          Show
          Jiaqi Tan added a comment - You can get away without root access, but say you were on a shared environment, e.g. you are taking a class on Hadoop at school and you are using a shared server, and everyone logs onto a (handful of, at most) gateway machine to submit jobs to a shared cluster, and you wanted to crunch through your logs using Chukwa, you wouldn't want to sit around waiting 5 minutes for the collector to commit the chunks, or wait for the Demux to get run periodically, if you just wanted to look at the logs of one job. Also, more importantly, you wouldn't be able to run agents+adaptors on all of the slave nodes, and you would need to set up MySQL (wouldn't be easy if you're a not-very-systemsy, average user), and do a lot of additional setup steps that the average Hadoop user wouldn't want to invest the effort just to look at the analysis of logs from one job. The aim is to lower the amount of effort needed to use Chukwa for log/metric analysis and anomaly detection, visualization, etc.
          Hide
          Ari Rabkin added a comment -

          Which parts of Chukwa currently require root access?

          Show
          Ari Rabkin added a comment - Which parts of Chukwa currently require root access?

            People

            • Assignee:
              Unassigned
              Reporter:
              Jiaqi Tan
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:

                Development