Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-216

postProcess.sh should locate hadoop-site.xml in HADOOP_CONF_DIR

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.1.2, 0.2.0
    • Fix Version/s: 0.2.0
    • Component/s: Data Processors
    • Labels:
      None
    • Environment:

      Redhat EL 5.1, Java 6

      Description

      postProcess.sh is currently hard coded to CHUKWA_HOME/conf as class path for locating configuration.
      postProcessManager should locate name node address and job tracker address from HADOOP_CONF_DIR or class path.

      1. CHUKWA-216.patch
        6 kB
        Jerome Boulon

        Activity

        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #45 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/45/ )
        Hide
        hudson Hudson added a comment -

        Integrated in Chukwa-trunk #43 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/43/)
        . postProcess.sh should use HADOOP_CONF_DIR to find hadoop-site.xml. Contributed by Jerome Boulon

        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #43 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/43/ ) . postProcess.sh should use HADOOP_CONF_DIR to find hadoop-site.xml. Contributed by Jerome Boulon
        Hide
        asrabkin Ari Rabkin added a comment -

        I just committed this to TRUNK. Thanks, Jerome!

        Show
        asrabkin Ari Rabkin added a comment - I just committed this to TRUNK. Thanks, Jerome!
        Hide
        eyang Eric Yang added a comment -

        +1 Looks good

        Show
        eyang Eric Yang added a comment - +1 Looks good
        Hide
        eyang Eric Yang added a comment -

        Remove this issue as a blocker for 0.1.2 release.

        Show
        eyang Eric Yang added a comment - Remove this issue as a blocker for 0.1.2 release.
        Hide
        asrabkin Ari Rabkin added a comment -

        This looks like a good idea. Have you folks tested this operationally?

        Show
        asrabkin Ari Rabkin added a comment - This looks like a good idea. Have you folks tested this operationally?
        Hide
        jboulon Jerome Boulon added a comment -

        + add HADOOP_CONF_DIR to the classpath

        Show
        jboulon Jerome Boulon added a comment - + add HADOOP_CONF_DIR to the classpath
        Hide
        jboulon Jerome Boulon added a comment -
        • Remove the add resoure for Hadoop and let HadoopConfiguration do the default, so for .20:
          --> addDefaultResource("core-default.xml");
          --> addDefaultResource("core-site.xml");
        Show
        jboulon Jerome Boulon added a comment - Remove the add resoure for Hadoop and let HadoopConfiguration do the default, so for .20: --> addDefaultResource("core-default.xml"); --> addDefaultResource("core-site.xml");
        Hide
        eyang Eric Yang added a comment -

        This code should be deprecated because it's incompatible going forward with newer version of hadoop. The proper fix is to ensure HADOOP_CONF_DIR as part of the class path, hence, we do not need to maintain any change for hadoop configuration change. namenode configuration should be added by the Configuration object itself rather than source hadoop-site.xml externally for all backend services.

        Show
        eyang Eric Yang added a comment - This code should be deprecated because it's incompatible going forward with newer version of hadoop. The proper fix is to ensure HADOOP_CONF_DIR as part of the class path, hence, we do not need to maintain any change for hadoop configuration change. namenode configuration should be added by the Configuration object itself rather than source hadoop-site.xml externally for all backend services.
        Hide
        jboulon Jerome Boulon added a comment -

        >> postProcessManager should locate name node address and job tracker address from HADOOP_CONF_DIR or class path.

        PostProcessManager is currently getting the configuration from ChukwaConfiguration class and loading hadoop-site.xml from the correct location.
        So could you be more specific on what's problem you are trying to solve?

        // Hadoop config
        String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
        if (hadoopConfDir != null)

        { super.addResource(new Path(hadoopConfDir + "/hadoop-site.xml")); log.debug("added " + hadoopConfDir + "/hadoop-site.xml to ChukwaConfiguration"); }

        else {
        String hadoopHome = System.getenv("HADOOP_HOME");
        if (hadoopHome != null)

        { super.addResource(new Path(hadoopHome + "/conf/hadoop-site.xml")); log.debug("added " + hadoopHome + "/hadoop-site.xml to ChukwaConfiguration"); }

        }

        Show
        jboulon Jerome Boulon added a comment - >> postProcessManager should locate name node address and job tracker address from HADOOP_CONF_DIR or class path. PostProcessManager is currently getting the configuration from ChukwaConfiguration class and loading hadoop-site.xml from the correct location. So could you be more specific on what's problem you are trying to solve? // Hadoop config String hadoopConfDir = System.getenv("HADOOP_CONF_DIR"); if (hadoopConfDir != null) { super.addResource(new Path(hadoopConfDir + "/hadoop-site.xml")); log.debug("added " + hadoopConfDir + "/hadoop-site.xml to ChukwaConfiguration"); } else { String hadoopHome = System.getenv("HADOOP_HOME"); if (hadoopHome != null) { super.addResource(new Path(hadoopHome + "/conf/hadoop-site.xml")); log.debug("added " + hadoopHome + "/hadoop-site.xml to ChukwaConfiguration"); } }
        Hide
        jboulon Jerome Boulon added a comment - - edited

        >>PostProcess.sh is currently hard coded to CHUKWA_HOME/conf as class path for locating configuration.
        This is a duplicate of CHUKWA-215

        Show
        jboulon Jerome Boulon added a comment - - edited >>PostProcess.sh is currently hard coded to CHUKWA_HOME/conf as class path for locating configuration. This is a duplicate of CHUKWA-215

          People

          • Assignee:
            jboulon Jerome Boulon
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development