Hadoop Common
  1. Hadoop Common
  2. HADOOP-7707

improve config generator to allow users to specify proxy user, turn append on or off, turn webhdfs on or off

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0, 0.23.0
    • Fix Version/s: 0.20.205.0, 0.23.0
    • Component/s: conf
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Added toggle for dfs.support.append, webhdfs and hadoop proxy user to setup config script. (Arpit Gupta via Eric Yang)
    1. HADOOP-7707.20s.patch
      5 kB
      Arpit Gupta
    2. HADOOP-7707.20s-1.patch
      5 kB
      Arpit Gupta
    3. HADOOP-7707.20s-2.patch
      7 kB
      Arpit Gupta
    4. HADOOP-7707.20s-3.patch
      5 kB
      Arpit Gupta
    5. HADOOP-7707.patch
      6 kB
      Arpit Gupta
    6. HADOOP-7707-1.patch
      6 kB
      Arpit Gupta

      Activity

      Hide
      Arpit Gupta added a comment -

      webhdfs needs the following configs

      dfs.webhdfs.enabled - true or false (defaults to false)

      If security is enabled the following configs will also be needed
      dfs.web.authentication.kerberos.principal - HTTP/_HOST@$REALM
      dfs.web.authentication.kerberos.keytab - path to the keytab file

      Show
      Arpit Gupta added a comment - webhdfs needs the following configs dfs.webhdfs.enabled - true or false (defaults to false) If security is enabled the following configs will also be needed dfs.web.authentication.kerberos.principal - HTTP/_HOST@$REALM dfs.web.authentication.kerberos.keytab - path to the keytab file
      Hide
      Arpit Gupta added a comment -

      For append

      dfs.support.append -> true or false (defaults to false)

      Show
      Arpit Gupta added a comment - For append dfs.support.append -> true or false (defaults to false)
      Hide
      Arpit Gupta added a comment -

      For proxy users will have to provide us which user (USER_VAL) needs to proxy and support groups and hosts for them.

      hadoop.proxyuser.$

      {USER_VAL}.groups -> values users,admin etc
      hadoop.proxyuser.${USER_VAL}

      .hosts -> hostname,ip etc

      Show
      Arpit Gupta added a comment - For proxy users will have to provide us which user (USER_VAL) needs to proxy and support groups and hosts for them. hadoop.proxyuser.$ {USER_VAL}.groups -> values users,admin etc hadoop.proxyuser.${USER_VAL} .hosts -> hostname,ip etc
      Hide
      Arpit Gupta added a comment -

      initial patch for branch 20 security

      Show
      Arpit Gupta added a comment - initial patch for branch 20 security
      Hide
      Arpit Gupta added a comment -

      the above patch has some issues, working on fixing the bugs

      Show
      Arpit Gupta added a comment - the above patch has some issues, working on fixing the bugs
      Hide
      Arpit Gupta added a comment -

      updated patch for 20 security, tested on a deployed cluster and able to generate the new configs values

      Show
      Arpit Gupta added a comment - updated patch for 20 security, tested on a deployed cluster and able to generate the new configs values
      Hide
      Arpit Gupta added a comment -

      Updated patch for branch 20.
      Added description to the new properties, also there were some statements missing in the setup conf when auto setup was not enabled.

      Show
      Arpit Gupta added a comment - Updated patch for branch 20. Added description to the new properties, also there were some statements missing in the setup conf when auto setup was not enabled.
      Hide
      Arpit Gupta added a comment -

      patch for trunk

      Show
      Arpit Gupta added a comment - patch for trunk
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12497368/HADOOP-7707.patch
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      +1 core tests. The patch passed unit tests in .

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/253//testReport/
      Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/253//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497368/HADOOP-7707.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/253//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/253//console This message is automatically generated.
      Hide
      Arpit Gupta added a comment -

      no tests included as this is part of deployment scripts, manually ran the script on a cluster to generate the new confs

      Show
      Arpit Gupta added a comment - no tests included as this is part of deployment scripts, manually ran the script on a cluster to generate the new confs
      Hide
      Arpit Gupta added a comment -

      updated the patch as the patch got stale

      Show
      Arpit Gupta added a comment - updated the patch as the patch got stale
      Hide
      Arpit Gupta added a comment -

      updated the patch as it got stale

      Show
      Arpit Gupta added a comment - updated the patch as it got stale
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12497718/HADOOP-7707-1.patch
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      +1 core tests. The patch passed unit tests in .

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/266//testReport/
      Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/266//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497718/HADOOP-7707-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/266//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/266//console This message is automatically generated.
      Hide
      Eric Yang added a comment -

      +1 verified it works.

      Show
      Eric Yang added a comment - +1 verified it works.
      Hide
      Eric Yang added a comment -

      I just committed this, thanks Arpit.

      Show
      Eric Yang added a comment - I just committed this, thanks Arpit.
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-trunk-Commit #1092 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1092/)
      HADOOP-7707. Added toggle for dfs.support.append, webhdfs and hadoop proxy
      user to setup config script. (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179001
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1092 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1092/ ) HADOOP-7707 . Added toggle for dfs.support.append, webhdfs and hadoop proxy user to setup config script. (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179001 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Common-trunk-Commit #1014 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1014/)
      HADOOP-7707. Added toggle for dfs.support.append, webhdfs and hadoop proxy
      user to setup config script. (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179001
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1014 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1014/ ) HADOOP-7707 . Added toggle for dfs.support.append, webhdfs and hadoop proxy user to setup config script. (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179001 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk-Commit #1032 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1032/)
      HADOOP-7707. Added toggle for dfs.support.append, webhdfs and hadoop proxy
      user to setup config script. (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179001
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1032 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1032/ ) HADOOP-7707 . Added toggle for dfs.support.append, webhdfs and hadoop proxy user to setup config script. (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179001 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-trunk #821 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/821/)
      HADOOP-7707. Added toggle for dfs.support.append, webhdfs and hadoop proxy
      user to setup config script. (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179001
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #821 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/821/ ) HADOOP-7707 . Added toggle for dfs.support.append, webhdfs and hadoop proxy user to setup config script. (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179001 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-0.23-Build #30 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/30/)
      HADOOP-7707. Added toggle for dfs.support.append, webhdfs and hadoop proxy
      user to setup config script. (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179002
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #30 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/30/ ) HADOOP-7707 . Added toggle for dfs.support.append, webhdfs and hadoop proxy user to setup config script. (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179002 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-0.23-Build #37 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/37/)
      HADOOP-7707. Added toggle for dfs.support.append, webhdfs and hadoop proxy
      user to setup config script. (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179002
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #37 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/37/ ) HADOOP-7707 . Added toggle for dfs.support.append, webhdfs and hadoop proxy user to setup config script. (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179002 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk #851 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/851/)
      HADOOP-7707. Added toggle for dfs.support.append, webhdfs and hadoop proxy
      user to setup config script. (Arpit Gupta via Eric Yang)

      eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179001
      Files :

      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh
      • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #851 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/851/ ) HADOOP-7707 . Added toggle for dfs.support.append, webhdfs and hadoop proxy user to setup config script. (Arpit Gupta via Eric Yang) eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179001 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/hadoop-setup-conf.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
      Hide
      Matt Foley added a comment -

      Confirmed with Eric that the second commit on Wed Oct 05 00:33:20 UTC 2011 was indeed also for HADOOP-7707, because some files were missed in the initial commit. Thanks, Eric!

      Show
      Matt Foley added a comment - Confirmed with Eric that the second commit on Wed Oct 05 00:33:20 UTC 2011 was indeed also for HADOOP-7707 , because some files were missed in the initial commit. Thanks, Eric!
      Hide
      Matt Foley added a comment -

      Closed upon release of 0.20.205.0

      Show
      Matt Foley added a comment - Closed upon release of 0.20.205.0

        People

        • Assignee:
          Arpit Gupta
          Reporter:
          Arpit Gupta
        • Votes:
          0 Vote for this issue
          Watchers:
          1 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development