Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-302

Chukwa Agent unable to start up and running

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.2.0
    • Fix Version/s: 0.2.0
    • Component/s: Data Collection
    • Labels:
      None
    • Environment:

      Redhat EL 5.1, Java 6

      Description

      agent.sh is hard coded with HADOOP_20_JAR. However, this may not reflect the correct running version of the hadoop library. agent.sh should obey HADOOP_JAR variable.

      1. CHUKWA-302.patch
        1 kB
        Eric Yang
      2. CHUKWA-302-1.patch
        3 kB
        Eric Yang

        Activity

        Hide
        eyang Eric Yang added a comment -

        Remove dependency of HADOOP_20_JAR, and HADOOP_18_JAR variables.

        Show
        eyang Eric Yang added a comment - Remove dependency of HADOOP_20_JAR, and HADOOP_18_JAR variables.
        Hide
        asrabkin Ari Rabkin added a comment -

        +1

        Show
        asrabkin Ari Rabkin added a comment - +1
        Hide
        jboulon Jerome Boulon added a comment -

        -1 in order to get an agreement not on the patch itself:

        What about if I'm running a bunch of web servers and I just want to collect the HTTP logs and I don't have hadoop at all?
        I understand that for the backend (processing pipeline) we don't want to include hadoop jars since we know that there's one already available somewhere but why do we assume that hadoop is installed on the system that we want to monitor?

        That bring back the packaging issue for the processing pipeline and the agent.
        Ps: this is a corner case that is not included in CHUKWA-297

        Show
        jboulon Jerome Boulon added a comment - -1 in order to get an agreement not on the patch itself: What about if I'm running a bunch of web servers and I just want to collect the HTTP logs and I don't have hadoop at all? I understand that for the backend (processing pipeline) we don't want to include hadoop jars since we know that there's one already available somewhere but why do we assume that hadoop is installed on the system that we want to monitor? That bring back the packaging issue for the processing pipeline and the agent. Ps: this is a corner case that is not included in CHUKWA-297
        Hide
        asrabkin Ari Rabkin added a comment -

        Users can and should be advised to have separate conf dirs for agents and collectors, if they need different hadoop versions.

        Show
        asrabkin Ari Rabkin added a comment - Users can and should be advised to have separate conf dirs for agents and collectors, if they need different hadoop versions.
        Hide
        asrabkin Ari Rabkin added a comment -

        Jerome, I don't understand the connection between the monitoring-non-hadoop and this patch. It's good to remind ourselves that Chukwa agents might run on a machine without Hadoop. But having a separate env variable for HADOOP_20 is no help, there.

        Your comment seems more germane to CHUKWA-297?

        Show
        asrabkin Ari Rabkin added a comment - Jerome, I don't understand the connection between the monitoring-non-hadoop and this patch. It's good to remind ourselves that Chukwa agents might run on a machine without Hadoop. But having a separate env variable for HADOOP_20 is no help, there. Your comment seems more germane to CHUKWA-297 ?
        Hide
        eyang Eric Yang added a comment -

        The current patch will fall back to CHUKWA_HOME/hadoopjar for the hadoop jar file. Since the hadoopjar in CHUKWA_HOME is package and supplied by chukwa at build time, it should always be compatible. If it's standalone web server, it is unlikely user would have HADOOP_HOME or HADOOP_JAR defined.

        In the remote case which the user has HADOOP_HOME defined in chukwa-env.shand cause the agent to break due to version incompatibility. We could add some extra wording in chukwa-env.sh to prevent this case from happening.

        Show
        eyang Eric Yang added a comment - The current patch will fall back to CHUKWA_HOME/hadoopjar for the hadoop jar file. Since the hadoopjar in CHUKWA_HOME is package and supplied by chukwa at build time, it should always be compatible. If it's standalone web server, it is unlikely user would have HADOOP_HOME or HADOOP_JAR defined. In the remote case which the user has HADOOP_HOME defined in chukwa-env.shand cause the agent to break due to version incompatibility. We could add some extra wording in chukwa-env.sh to prevent this case from happening.
        Hide
        eyang Eric Yang added a comment -

        Added new warning message about HADOOP_JAR and how to use it.

        Show
        eyang Eric Yang added a comment - Added new warning message about HADOOP_JAR and how to use it.
        Hide
        asrabkin Ari Rabkin added a comment -

        I would commit this. It's a large step forward.

        Also. I think we ought to have a paragraph in the chukwa admin guide about how chukwa finds a hadoop, and what the right thing to do is. Can do a separate JIRA for that.

        Show
        asrabkin Ari Rabkin added a comment - I would commit this. It's a large step forward. Also. I think we ought to have a paragraph in the chukwa admin guide about how chukwa finds a hadoop, and what the right thing to do is. Can do a separate JIRA for that.
        Hide
        eyang Eric Yang added a comment -

        I just committed this, thanks Ari and Jerome.

        Show
        eyang Eric Yang added a comment - I just committed this, thanks Ari and Jerome.
        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #61 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/61/ )

          People

          • Assignee:
            eyang Eric Yang
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development