Chukwa
  1. Chukwa
  2. CHUKWA-216

postProcess.sh should locate hadoop-site.xml in HADOOP_CONF_DIR

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.1.2, 0.2.0
    • Fix Version/s: 0.2.0
    • Component/s: Data Processors
    • Labels:
      None
    • Environment:

      Redhat EL 5.1, Java 6

      Description

      postProcess.sh is currently hard coded to CHUKWA_HOME/conf as class path for locating configuration.
      postProcessManager should locate name node address and job tracker address from HADOOP_CONF_DIR or class path.

      1. CHUKWA-216.patch
        6 kB
        Jerome Boulon

        Activity

        Hide
        Jerome Boulon added a comment - - edited

        >>PostProcess.sh is currently hard coded to CHUKWA_HOME/conf as class path for locating configuration.
        This is a duplicate of CHUKWA-215

        Show
        Jerome Boulon added a comment - - edited >>PostProcess.sh is currently hard coded to CHUKWA_HOME/conf as class path for locating configuration. This is a duplicate of CHUKWA-215
        Hide
        Jerome Boulon added a comment -

        >> postProcessManager should locate name node address and job tracker address from HADOOP_CONF_DIR or class path.

        PostProcessManager is currently getting the configuration from ChukwaConfiguration class and loading hadoop-site.xml from the correct location.
        So could you be more specific on what's problem you are trying to solve?

        // Hadoop config
        String hadoopConfDir = System.getenv("HADOOP_CONF_DIR");
        if (hadoopConfDir != null)

        { super.addResource(new Path(hadoopConfDir + "/hadoop-site.xml")); log.debug("added " + hadoopConfDir + "/hadoop-site.xml to ChukwaConfiguration"); }

        else {
        String hadoopHome = System.getenv("HADOOP_HOME");
        if (hadoopHome != null)

        { super.addResource(new Path(hadoopHome + "/conf/hadoop-site.xml")); log.debug("added " + hadoopHome + "/hadoop-site.xml to ChukwaConfiguration"); }

        }

        Show
        Jerome Boulon added a comment - >> postProcessManager should locate name node address and job tracker address from HADOOP_CONF_DIR or class path. PostProcessManager is currently getting the configuration from ChukwaConfiguration class and loading hadoop-site.xml from the correct location. So could you be more specific on what's problem you are trying to solve? // Hadoop config String hadoopConfDir = System.getenv("HADOOP_CONF_DIR"); if (hadoopConfDir != null) { super.addResource(new Path(hadoopConfDir + "/hadoop-site.xml")); log.debug("added " + hadoopConfDir + "/hadoop-site.xml to ChukwaConfiguration"); } else { String hadoopHome = System.getenv("HADOOP_HOME"); if (hadoopHome != null) { super.addResource(new Path(hadoopHome + "/conf/hadoop-site.xml")); log.debug("added " + hadoopHome + "/hadoop-site.xml to ChukwaConfiguration"); } }
        Hide
        Eric Yang added a comment -

        This code should be deprecated because it's incompatible going forward with newer version of hadoop. The proper fix is to ensure HADOOP_CONF_DIR as part of the class path, hence, we do not need to maintain any change for hadoop configuration change. namenode configuration should be added by the Configuration object itself rather than source hadoop-site.xml externally for all backend services.

        Show
        Eric Yang added a comment - This code should be deprecated because it's incompatible going forward with newer version of hadoop. The proper fix is to ensure HADOOP_CONF_DIR as part of the class path, hence, we do not need to maintain any change for hadoop configuration change. namenode configuration should be added by the Configuration object itself rather than source hadoop-site.xml externally for all backend services.
        Hide
        Jerome Boulon added a comment -
        • Remove the add resoure for Hadoop and let HadoopConfiguration do the default, so for .20:
          --> addDefaultResource("core-default.xml");
          --> addDefaultResource("core-site.xml");
        Show
        Jerome Boulon added a comment - Remove the add resoure for Hadoop and let HadoopConfiguration do the default, so for .20: --> addDefaultResource("core-default.xml"); --> addDefaultResource("core-site.xml");
        Hide
        Jerome Boulon added a comment -

        + add HADOOP_CONF_DIR to the classpath

        Show
        Jerome Boulon added a comment - + add HADOOP_CONF_DIR to the classpath
        Hide
        Ari Rabkin added a comment -

        This looks like a good idea. Have you folks tested this operationally?

        Show
        Ari Rabkin added a comment - This looks like a good idea. Have you folks tested this operationally?
        Hide
        Eric Yang added a comment -

        Remove this issue as a blocker for 0.1.2 release.

        Show
        Eric Yang added a comment - Remove this issue as a blocker for 0.1.2 release.
        Hide
        Eric Yang added a comment -

        +1 Looks good

        Show
        Eric Yang added a comment - +1 Looks good
        Hide
        Ari Rabkin added a comment -

        I just committed this to TRUNK. Thanks, Jerome!

        Show
        Ari Rabkin added a comment - I just committed this to TRUNK. Thanks, Jerome!
        Hide
        Hudson added a comment -

        Integrated in Chukwa-trunk #43 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/43/)
        . postProcess.sh should use HADOOP_CONF_DIR to find hadoop-site.xml. Contributed by Jerome Boulon

        Show
        Hudson added a comment - Integrated in Chukwa-trunk #43 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/43/ ) . postProcess.sh should use HADOOP_CONF_DIR to find hadoop-site.xml. Contributed by Jerome Boulon
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Chukwa-trunk #45 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/45/ )

          People

          • Assignee:
            Jerome Boulon
            Reporter:
            Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development