Hadoop Common
  1. Hadoop Common
  2. HADOOP-7599

Improve hadoop setup conf script to setup secure Hadoop cluster

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.203.0
    • Fix Version/s: 0.20.205.0, 0.23.0, 0.24.0
    • Component/s: scripts
    • Labels:
      None
    • Environment:

      Java 6, RHEL 5.6

      Description

      Setting up a secure Hadoop cluster requires a lot of manual setup. The motivation of this jira is to provide setup scripts to automate setup secure Hadoop cluster.

      1. HADOOP-7599.patch
        54 kB
        Eric Yang
      2. HADOOP-7599-1.patch
        56 kB
        Eric Yang
      3. HADOOP-7599-2.patch
        56 kB
        Eric Yang
      4. HADOOP-7599-3.patch
        56 kB
        Eric Yang
      5. HADOOP-7599-4.patch
        68 kB
        Eric Yang
      6. HADOOP-7599-5.patch
        69 kB
        Eric Yang
      7. HADOOP-7599-trunk.patch
        70 kB
        Eric Yang
      8. HADOOP-7599-trunk-2.patch
        70 kB
        Eric Yang
      9. HADOOP-7599-trunk-3.patch
        70 kB
        Eric Yang
      10. HADOOP-7599-trunk-4.patch
        82 kB
        Eric Yang
      11. HADOOP-7599-trunk-5.patch
        82 kB
        Eric Yang

        Issue Links

          Activity

          Hide
          Matt Foley added a comment -

          Closed upon release of 0.20.205.0

          Show
          Matt Foley added a comment - Closed upon release of 0.20.205.0
          Hide
          Eric Yang added a comment -

          Aaron, inclusion of the files are intentional. However, we probably want to put license on the files. I filed HADOOP-7641 to track the license changes. Thanks

          Show
          Eric Yang added a comment - Aaron, inclusion of the files are intentional. However, we probably want to put license on the files. I filed HADOOP-7641 to track the license changes. Thanks
          Hide
          Aaron T. Myers added a comment -

          I think that by including the contents of the templates/ directories, this patch introduced several release audit warnings. See this for pre-commit Jenkins run for an example.

          Was the inclusion of these files intentional? If so, should we just add the relevant directories to RAT's exclude list(s)?

          Show
          Aaron T. Myers added a comment - I think that by including the contents of the templates/ directories, this patch introduced several release audit warnings. See this for pre-commit Jenkins run for an example. Was the inclusion of these files intentional? If so, should we just add the relevant directories to RAT's exclude list(s)?
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #792 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/792/)
          HADOOP-7599. Script improvements to setup a secure Hadoop cluster. Contributed by Eric Yang.

          ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169986
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-datanode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-jobtracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-namenode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-tasktracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-create-user.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-hdfs.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-single-node.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-datanode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-jobtracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-namenode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-tasktracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/capacity-scheduler.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/commons-logging.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/core-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-policy.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-queue-acls.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/taskcontroller.cfg
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #792 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/792/ ) HADOOP-7599 . Script improvements to setup a secure Hadoop cluster. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169986 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-datanode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-jobtracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-namenode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-tasktracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-create-user.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-hdfs.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-single-node.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-datanode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-jobtracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-namenode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-tasktracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/capacity-scheduler.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/commons-logging.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/core-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-policy.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-queue-acls.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/taskcontroller.cfg
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #816 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/816/)
          HADOOP-7599. Script improvements to setup a secure Hadoop cluster. Contributed by Eric Yang.

          ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169986
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-datanode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-jobtracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-namenode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-tasktracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-create-user.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-hdfs.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-single-node.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-datanode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-jobtracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-namenode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-tasktracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/capacity-scheduler.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/commons-logging.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/core-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-policy.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-queue-acls.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/taskcontroller.cfg
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #816 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/816/ ) HADOOP-7599 . Script improvements to setup a secure Hadoop cluster. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169986 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-datanode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-jobtracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-namenode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-tasktracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-create-user.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-hdfs.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-single-node.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-datanode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-jobtracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-namenode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-tasktracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/capacity-scheduler.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/commons-logging.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/core-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-policy.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-queue-acls.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/taskcontroller.cfg
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #890 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/890/)
          HADOOP-7599. Script improvements to setup a secure Hadoop cluster. Contributed by Eric Yang.

          ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169986
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-datanode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-jobtracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-namenode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-tasktracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-create-user.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-hdfs.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-single-node.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-datanode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-jobtracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-namenode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-tasktracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/capacity-scheduler.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/commons-logging.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/core-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-policy.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-queue-acls.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/taskcontroller.cfg
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #890 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/890/ ) HADOOP-7599 . Script improvements to setup a secure Hadoop cluster. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169986 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-datanode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-jobtracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-namenode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-tasktracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-create-user.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-hdfs.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-single-node.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-datanode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-jobtracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-namenode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-tasktracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/capacity-scheduler.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/commons-logging.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/core-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-policy.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-queue-acls.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/taskcontroller.cfg
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #955 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/955/)
          HADOOP-7599. Script improvements to setup a secure Hadoop cluster. Contributed by Eric Yang.

          ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169986
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-datanode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-jobtracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-namenode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-tasktracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-create-user.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-hdfs.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-single-node.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-datanode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-jobtracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-namenode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-tasktracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/capacity-scheduler.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/commons-logging.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/core-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-policy.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-queue-acls.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/taskcontroller.cfg
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #955 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/955/ ) HADOOP-7599 . Script improvements to setup a secure Hadoop cluster. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169986 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-datanode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-jobtracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-namenode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-tasktracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-create-user.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-hdfs.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-single-node.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-datanode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-jobtracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-namenode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-tasktracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/capacity-scheduler.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/commons-logging.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/core-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-policy.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-queue-acls.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/taskcontroller.cfg
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #878 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/878/)
          HADOOP-7599. Script improvements to setup a secure Hadoop cluster. Contributed by Eric Yang.

          ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169986
          Files :

          • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-datanode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-jobtracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-namenode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-tasktracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-create-user.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-hdfs.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-single-node.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-datanode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-jobtracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-namenode
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-tasktracker
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/capacity-scheduler.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/commons-logging.properties
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/core-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-policy.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-queue-acls.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/taskcontroller.cfg
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #878 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/878/ ) HADOOP-7599 . Script improvements to setup a secure Hadoop cluster. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169986 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-datanode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-jobtracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-namenode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/deb/init.d/hadoop-tasktracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-create-user.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-hdfs.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-single-node.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-datanode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-jobtracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-namenode /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/rpm/init.d/hadoop-tasktracker /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/capacity-scheduler.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/commons-logging.properties /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/core-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-policy.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-queue-acls.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/taskcontroller.cfg
          Hide
          Devaraj Das added a comment -

          Committed the patch on 0.23 and trunk. Thanks, Eric!

          Show
          Devaraj Das added a comment - Committed the patch on 0.23 and trunk. Thanks, Eric!
          Hide
          Devaraj Das added a comment -

          I am going to commit the patch in 0.23. When Yarn stabilizes, we can raise another jira and do the appropriate fixes.

          Show
          Devaraj Das added a comment - I am going to commit the patch in 0.23. When Yarn stabilizes, we can raise another jira and do the appropriate fixes.
          Hide
          Devaraj Das added a comment -

          I committed the patch on branch-0.20-security. I am hesitant to commit the patch on trunk/0.23 yet since this patch needs some work to make Yarn setup work. Lets discuss..

          Show
          Devaraj Das added a comment - I committed the patch on branch-0.20-security. I am hesitant to commit the patch on trunk/0.23 yet since this patch needs some work to make Yarn setup work. Lets discuss..
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493942/HADOOP-7599-5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/161//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493942/HADOOP-7599-5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/161//console This message is automatically generated.
          Hide
          Eric Yang added a comment -

          Bugfix to make sure non-secure Hadoop cluster also works.

          Show
          Eric Yang added a comment - Bugfix to make sure non-secure Hadoop cluster also works.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493931/HADOOP-7599-trunk-4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/160//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493931/HADOOP-7599-trunk-4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/160//console This message is automatically generated.
          Hide
          Eric Yang added a comment -
          • Revised hadoop-create-user.sh to include ability to create user by authorized super user keytab.
          • Revised hadoop-env.sh for settting effective user permission properly.
          Show
          Eric Yang added a comment - Revised hadoop-create-user.sh to include ability to create user by authorized super user keytab. Revised hadoop-env.sh for settting effective user permission properly.
          Hide
          Eric Yang added a comment -

          In the patch, I put in the configuration properties that are not security related but are good configurations that are tested to work in large 0.20.2xx clusters.

          mapred.heartbeats.in.second - enable HADOOP:5784
          mapreduce.tasktracker.outofband.heartbeat - enable MAPREDUCE:270
          mapred.jobtracker.maxtasks.per.job - safe guard job tracker from running out of memory
          mapred.jobtracker.retirejob.check - 10000
          mapred.jobtracker.retirejob.interval - 0
          mapred.map.tasks.speculative.execution - false
          mapred.reduce.tasks.speculative.execution - false
          mapred.tasktracker.tasks.sleeptime-before-sigkill - 250

          Show
          Eric Yang added a comment - In the patch, I put in the configuration properties that are not security related but are good configurations that are tested to work in large 0.20.2xx clusters. mapred.heartbeats.in.second - enable HADOOP:5784 mapreduce.tasktracker.outofband.heartbeat - enable MAPREDUCE:270 mapred.jobtracker.maxtasks.per.job - safe guard job tracker from running out of memory mapred.jobtracker.retirejob.check - 10000 mapred.jobtracker.retirejob.interval - 0 mapred.map.tasks.speculative.execution - false mapred.reduce.tasks.speculative.execution - false mapred.tasktracker.tasks.sleeptime-before-sigkill - 250
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493878/HADOOP-7599-3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/154//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493878/HADOOP-7599-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/154//console This message is automatically generated.
          Hide
          Eric Yang added a comment -

          Update reference to task tracker group.

          Show
          Eric Yang added a comment - Update reference to task tracker group.
          Hide
          Devaraj Das added a comment -

          Please remove the config properties that aren't generated by the scripts in this patch (bullet 12 in my first comment). I also noticed now that there are references to 'hadoop' group still present. Please replace them with the variable you defined for the special group.

          Show
          Devaraj Das added a comment - Please remove the config properties that aren't generated by the scripts in this patch (bullet 12 in my first comment). I also noticed now that there are references to 'hadoop' group still present. Please replace them with the variable you defined for the special group.
          Hide
          Matt Foley added a comment -

          Hi, I talked with Devaraj about item #9 in his comment above.

          I believe that for a simple setup what Eric is doing is okay, and I would like to have it in 205. Please go ahead and commit, and we can continue making it better in the next release, perhaps by using rpm and deb upgrade/update features as Devaraj mentioned. Thanks.

          Show
          Matt Foley added a comment - Hi, I talked with Devaraj about item #9 in his comment above . I believe that for a simple setup what Eric is doing is okay, and I would like to have it in 205. Please go ahead and commit, and we can continue making it better in the next release, perhaps by using rpm and deb upgrade/update features as Devaraj mentioned. Thanks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493865/HADOOP-7599-trunk-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/153//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493865/HADOOP-7599-trunk-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/153//console This message is automatically generated.
          Hide
          Eric Yang added a comment -
          • Make sure keytab file ownership are setup correctly.
          Show
          Eric Yang added a comment - Make sure keytab file ownership are setup correctly.
          Hide
          Devaraj Das added a comment -

          Went over the patch. Some comments:
          1. Don't chmod the keytab dir contents to 755. The keytab files should be owned by the user running the respective daemon, and 700ed.
          2. On the bullet#9 in my last comment, you can do a check for empty config files (like if the strings '<property>' and/or '<value>' occurs, the config file is not empty). Not pretty but safer.. Long term, Hadoop could stop shipping the empty config files.

          Show
          Devaraj Das added a comment - Went over the patch. Some comments: 1. Don't chmod the keytab dir contents to 755. The keytab files should be owned by the user running the respective daemon, and 700ed. 2. On the bullet#9 in my last comment, you can do a check for empty config files (like if the strings '<property>' and/or '<value>' occurs, the config file is not empty). Not pretty but safer.. Long term, Hadoop could stop shipping the empty config files.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493816/HADOOP-7599-trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/151//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493816/HADOOP-7599-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/151//console This message is automatically generated.
          Hide
          Eric Yang added a comment -

          Same patch for trunk.

          Show
          Eric Yang added a comment - Same patch for trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493726/HADOOP-7599-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/150//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493726/HADOOP-7599-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/150//console This message is automatically generated.
          Hide
          Eric Yang added a comment -

          Addressing Devaraj's concerns in bulletin item 1,2,3,7,8,10,12.

          Show
          Eric Yang added a comment - Addressing Devaraj's concerns in bulletin item 1,2,3,7,8,10,12.
          Hide
          Eric Yang added a comment -

          6. Why did we remove the call to hadoop-setup-config.sh from src/packages/hadoop-setup-conf.sh

          Previously, hadoop-setup-conf.sh refers to $PREFIX/libexec/hadoop-config.sh to source existing HADOOP environment variables. It is safer to make hadoop-setup-conf.sh independent from existing environment to avoid recursively append of environment variables, i.e. HADOOP_LOG_DIR=/var/log/hadoop/root/root.

          Show
          Eric Yang added a comment - 6. Why did we remove the call to hadoop-setup-config.sh from src/packages/hadoop-setup-conf.sh Previously, hadoop-setup-conf.sh refers to $PREFIX/libexec/hadoop-config.sh to source existing HADOOP environment variables. It is safer to make hadoop-setup-conf.sh independent from existing environment to avoid recursively append of environment variables, i.e. HADOOP_LOG_DIR=/var/log/hadoop/root/root.
          Hide
          Eric Yang added a comment -
          1. I will polish this.
          2. I will polish this.
          3. I will polish this.
          4. --datanodes and --tasktrackers are already optional
          5. this is a bug fix. namenode-host is more widely used than namenode-url. It is better to make the name changes. This script was only working internally in RPM post installation script. Therefore, it does not break backward compatibility in this name changes.
          6. I don't understand this.
          7. I will polish this.
          8. I will polish this.
          9. By default, hadoop tar ball ships with empty configuration files. Consequencely, if --force flag is implemented, --force flag becomes mandatory in order to run the script. This is counter productive to have this flag because user will always use --force flag in order to run the setup script. User will take this for granted and --force loses it's ability to safe guard user from making mistakes.
          10. I will polish this.
          11. HADOOP_SECURE_DN_LOG_DIR is used in hadoop-daemon.sh
          12. I will polish this.
          Show
          Eric Yang added a comment - I will polish this. I will polish this. I will polish this. --datanodes and --tasktrackers are already optional this is a bug fix. namenode-host is more widely used than namenode-url. It is better to make the name changes. This script was only working internally in RPM post installation script. Therefore, it does not break backward compatibility in this name changes. I don't understand this. I will polish this. I will polish this. By default, hadoop tar ball ships with empty configuration files. Consequencely, if --force flag is implemented, --force flag becomes mandatory in order to run the script. This is counter productive to have this flag because user will always use --force flag in order to run the setup script. User will take this for granted and --force loses it's ability to safe guard user from making mistakes. I will polish this. HADOOP_SECURE_DN_LOG_DIR is used in hadoop-daemon.sh I will polish this.
          Hide
          Devaraj Das added a comment -

          Some comments:
          1. The mapred system dir should be 700ed
          2. The -format is always called by default in hadoop-setup-hdfs. Can we make the formatting done based on command line option (-format) provided by the user running the script. I really don't want implicit namenode formatting.
          3. We could let the directory /mapred be group owned by the system group.
          4. The --datanodes & --tasktrackers options - could they be made optional? They might be already optional. Please confirm.
          5. Could we live with namenode-url instead of replacing it with namenode-host (ditto for jobtracker-url). I also see you changed a couple of other places in a backward incompatible way (like HADOOP_JT_HOST, taskcontroller ownership). Want to avoid incompatible changes.
          6. Why did we remove the call to hadoop-setup-config.sh from src/packages/hadoop-setup-conf.sh
          7. The group is hardcoded to 'hadoop' in a couple of places. Can we avoid that?
          8. I see commented-out lines in hadoop-setup-conf.sh. Please remove those.
          9. I don't think we should blindly overwrite the conf files in the config directory. We probably should warn & exit if the conf directory already has some files within. The user can probably use --force if he wants to avoid the warning.
          10. All the components in the path leading up to taskcontroller.cfg has to be owned by root. Have this checked.
          11. Where is HADOOP_SECURE_DN_LOG_DIR used?
          12. A whole lot of configuration options that you added in the *-site.xml files are already there in *-default.xml. We don't need those. We only need the security related ones. Also, we don't want to play with the non-security configs like mapred.tasktracker.map.tasks.maximum.

          Show
          Devaraj Das added a comment - Some comments: 1. The mapred system dir should be 700ed 2. The -format is always called by default in hadoop-setup-hdfs. Can we make the formatting done based on command line option (-format) provided by the user running the script. I really don't want implicit namenode formatting. 3. We could let the directory /mapred be group owned by the system group. 4. The --datanodes & --tasktrackers options - could they be made optional? They might be already optional. Please confirm. 5. Could we live with namenode-url instead of replacing it with namenode-host (ditto for jobtracker-url). I also see you changed a couple of other places in a backward incompatible way (like HADOOP_JT_HOST, taskcontroller ownership). Want to avoid incompatible changes. 6. Why did we remove the call to hadoop-setup-config.sh from src/packages/hadoop-setup-conf.sh 7. The group is hardcoded to 'hadoop' in a couple of places. Can we avoid that? 8. I see commented-out lines in hadoop-setup-conf.sh. Please remove those. 9. I don't think we should blindly overwrite the conf files in the config directory. We probably should warn & exit if the conf directory already has some files within. The user can probably use --force if he wants to avoid the warning. 10. All the components in the path leading up to taskcontroller.cfg has to be owned by root. Have this checked. 11. Where is HADOOP_SECURE_DN_LOG_DIR used? 12. A whole lot of configuration options that you added in the *-site.xml files are already there in *-default.xml. We don't need those. We only need the security related ones. Also, we don't want to play with the non-security configs like mapred.tasktracker.map.tasks.maximum.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12492693/HADOOP-7599.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/122//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492693/HADOOP-7599.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/122//console This message is automatically generated.
          Hide
          Eric Yang added a comment -

          This patch depends on the improved classpath detection for binary or tar layout.

          Show
          Eric Yang added a comment - This patch depends on the improved classpath detection for binary or tar layout.
          Hide
          Eric Yang added a comment -

          Setup scripts for setting up secure Hadoop cluster.

          For usage of the scripts run:

          hadoop-setup-conf.sh -h
          hadoop-setup-hdfs.sh -h
          

          Prerequisite

          keytab files are pre-generated and placed in /etc/security/keytabs.

          Setup sequence

          # Run on all nodes
          sudo hadoop-setup-conf.sh --java-home=...
          # Run on nematode
          sudo hadoop-setup-hdfs --config ...
          # Run on datanodes
          sudo hadoop-daemon.sh --config $HADOOP_CONF_DIR start datanode
          # Run on jobtracker as mapreduce user
          hadoop-daemon.sh --config $HADOOP_CONF_DIR start jobtracker
          # Run on tasktracker as mapreduce user
          hadoop-daemon.sh --config $HADOOP_CONF_DIR start tasktracker
          
          Show
          Eric Yang added a comment - Setup scripts for setting up secure Hadoop cluster. For usage of the scripts run: hadoop-setup-conf.sh -h hadoop-setup-hdfs.sh -h Prerequisite keytab files are pre-generated and placed in /etc/security/keytabs. Setup sequence # Run on all nodes sudo hadoop-setup-conf.sh --java-home=... # Run on nematode sudo hadoop-setup-hdfs --config ... # Run on datanodes sudo hadoop-daemon.sh --config $HADOOP_CONF_DIR start datanode # Run on jobtracker as mapreduce user hadoop-daemon.sh --config $HADOOP_CONF_DIR start jobtracker # Run on tasktracker as mapreduce user hadoop-daemon.sh --config $HADOOP_CONF_DIR start tasktracker

            People

            • Assignee:
              Eric Yang
              Reporter:
              Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development