Bigtop
  1. Bigtop
  2. BIGTOP-105

hadoop services can not be started with the default configs in place

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.1.0
    • Fix Version/s: None
    • Component/s: General
    • Labels:
      None

      Description

      All of the services packaged by Bigtop provide some level of default configuration. Hadoop is the only exception where an extra package is needed to make services usable. I would like to change this by:
      1. Introducing the following snippets into the configs that get shipped with hadoop package by default:
      core-site.xml:
      <property>
      <name>fs.default.name</name>
      <value>hdfs://localhost:8020</value>
      </property>

      mapred-site.xml:
      <property>
      <name>mapred.job.tracker</name>
      <value>localhost:8021</value>
      </property>

      2. move formating of the namenode from hadoop-conf-pseudo into hadoop-namenode package

      1. BIGTOP-105.patch.txt
        1 kB
        Roman Shaposhnik
      2. BIGTOP-105.patch.txt
        10 kB
        Roman Shaposhnik

        Activity

        Hide
        Roman Shaposhnik added a comment -

        An alternative would be to make upstream hadoop have reasonable defaults so that services can be started with empty configs.

        Show
        Roman Shaposhnik added a comment - An alternative would be to make upstream hadoop have reasonable defaults so that services can be started with empty configs.
        Hide
        Eli Collins added a comment -

        It's worth looking into what's done upstream in src/packages so the conf is similar (see hadoop-setup-conf).

        Show
        Eli Collins added a comment - It's worth looking into what's done upstream in src/packages so the conf is similar (see hadoop-setup-conf).
        Hide
        Roman Shaposhnik added a comment -

        Eli, what's the proper way of generating a tarball with configs (IOW, a thing that would later become an Apache release of Hadoop .23)? I tried this:
        $ mvn package -DskipTests -Pdist -Dtar

        and the resulting tarball didn't have much configs at all:
        $ $ tar tvf ./hadoop-common-project/hadoop-common/target/hadoop-common-0.24.0-SNAPSHOT.tar.gz| fgrep "/etc"
        drwxr-xr-x rvs/rvs 0 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/
        drwxr-xr-x rvs/rvs 0 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/hadoop/
        rw-rr- rvs/rvs 2301 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/hadoop/hadoop-metrics.properties
        rw-rr- rvs/rvs 2376 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/hadoop/hadoop-metrics2.properties
        rw-rr- rvs/rvs 1243 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/hadoop/ssl-client.xml.example
        rw-rr- rvs/rvs 6354 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/hadoop/log4j.properties
        rw-rr- rvs/rvs 1195 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/hadoop/ssl-server.xml.example

        Show
        Roman Shaposhnik added a comment - Eli, what's the proper way of generating a tarball with configs (IOW, a thing that would later become an Apache release of Hadoop .23)? I tried this: $ mvn package -DskipTests -Pdist -Dtar and the resulting tarball didn't have much configs at all: $ $ tar tvf ./hadoop-common-project/hadoop-common/target/hadoop-common-0.24.0-SNAPSHOT.tar.gz| fgrep "/etc" drwxr-xr-x rvs/rvs 0 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/ drwxr-xr-x rvs/rvs 0 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/hadoop/ rw-r r - rvs/rvs 2301 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/hadoop/hadoop-metrics.properties rw-r r - rvs/rvs 2376 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/hadoop/hadoop-metrics2.properties rw-r r - rvs/rvs 1243 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/hadoop/ssl-client.xml.example rw-r r - rvs/rvs 6354 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/hadoop/log4j.properties rw-r r - rvs/rvs 1195 2011-09-22 11:59 hadoop-common-0.24.0-SNAPSHOT/etc/hadoop/ssl-server.xml.example
        Hide
        Patrick Hunt added a comment -

        FYI some details here I would think: http://wiki.apache.org/hadoop/HowToRelease

        Show
        Patrick Hunt added a comment - FYI some details here I would think: http://wiki.apache.org/hadoop/HowToRelease
        Hide
        Roman Shaposhnik added a comment -

        Those instructions seem to be pretty outdated by now. However, there seems to be a relevant JIRA: HADOOP-7673

        Show
        Roman Shaposhnik added a comment - Those instructions seem to be pretty outdated by now. However, there seems to be a relevant JIRA: HADOOP-7673
        Hide
        Roman Shaposhnik added a comment -

        This is, by no means, an actual solution. Just a workaround that we need to put in place in order to unblock Bigtop 0.2.0

        Show
        Roman Shaposhnik added a comment - This is, by no means, an actual solution. Just a workaround that we need to put in place in order to unblock Bigtop 0.2.0
        Hide
        Bruno Mahé added a comment -

        +1

        Show
        Bruno Mahé added a comment - +1
        Hide
        Roman Shaposhnik added a comment -

        Now that the workaround is in place – pushing the real fix out to 0.3.0

        Show
        Roman Shaposhnik added a comment - Now that the workaround is in place – pushing the real fix out to 0.3.0
        Hide
        Roman Shaposhnik added a comment -

        The current patch doesn't quite apply and it is not even clear we want to pursue this anymore.

        Show
        Roman Shaposhnik added a comment - The current patch doesn't quite apply and it is not even clear we want to pursue this anymore.

          People

          • Assignee:
            Roman Shaposhnik
            Reporter:
            Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development