Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8561

Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.3-alpha, 0.23.6, 1.1.2
    • Component/s: security
    • Labels:
      None
    • Target Version/s:

      Description

      To solve the problem for an authenticated user to type hadoop shell commands in a web console, we can introduce an HADOOP_PROXY_USER environment variable to allow proper impersonation in the child hadoop client processes.

      1. hadoop-8561-branch-1.patch
        4 kB
        Yu Gao
      2. hadoop-8561-branch-2.patch
        4 kB
        Yu Gao
      3. hadoop-8561.patch
        4 kB
        Yu Gao
      4. hadoop-8561-v2.patch
        4 kB
        Luke Lu

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #470 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/470/)
          HADOOP-8561. Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes (Yu Gao via tgraves) (Revision 1424698)

          Result = UNSTABLE
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424698
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #470 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/470/ ) HADOOP-8561 . Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes (Yu Gao via tgraves) (Revision 1424698) Result = UNSTABLE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424698 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1287 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1287/)
          HADOOP-8561. Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429)

          Result = SUCCESS
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1287 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1287/ ) HADOOP-8561 . Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1256 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1256/)
          HADOOP-8561. Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429)

          Result = FAILURE
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1256 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1256/ ) HADOOP-8561 . Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429) Result = FAILURE llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #67 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/67/)
          HADOOP-8561. Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429)

          Result = SUCCESS
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Yarn-trunk #67 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/67/ ) HADOOP-8561 . Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Hide
          vicaya Luke Lu added a comment -

          Committed to trunk and branch-

          {2,1,1.1}

          . Thanks Yu!

          Show
          vicaya Luke Lu added a comment - Committed to trunk and branch- {2,1,1.1} . Thanks Yu!
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3128 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3128/)
          HADOOP-8561. Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429)

          Result = SUCCESS
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-trunk-Commit #3128 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3128/ ) HADOOP-8561 . Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12561097/hadoop-8561-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1878//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1878//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12561097/hadoop-8561-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1878//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1878//console This message is automatically generated.
          Hide
          vicaya Luke Lu added a comment -

          Need to merge with HADOOP-9035

          Show
          vicaya Luke Lu added a comment - Need to merge with HADOOP-9035
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551912/hadoop-8561.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1701//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1701//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551912/hadoop-8561.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1701//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1701//console This message is automatically generated.
          Hide
          vicaya Luke Lu added a comment -

          The patches lgtm. +1 pending jenkins.

          Show
          vicaya Luke Lu added a comment - The patches lgtm. +1 pending jenkins.
          Hide
          vicaya Luke Lu added a comment -

          This approach has added benefit of working with clients (like HBase shell) not written in Java.

          Using an env makes me a bit squeamish since it may introduce an unexpected attack vector.

          It won't do anything for ordinary users. An admin web app of course needs to do a few things sanitize the input to disallow fork/exec etc.

          Show
          vicaya Luke Lu added a comment - This approach has added benefit of working with clients (like HBase shell) not written in Java. Using an env makes me a bit squeamish since it may introduce an unexpected attack vector. It won't do anything for ordinary users. An admin web app of course needs to do a few things sanitize the input to disallow fork/exec etc.
          Hide
          mattf Matt Foley added a comment -

          Moved to 1.2.0 upon release of 1.1.0.

          Show
          mattf Matt Foley added a comment - Moved to 1.2.0 upon release of 1.1.0.
          Hide
          daryn Daryn Sharp added a comment -

          I kind of like Todd's approach. Maybe we should consider adding a sudo command to FsShell so it's not a separate utility. Using an env makes me a bit squeamish since it may introduce an unexpected attack vector.

          Show
          daryn Daryn Sharp added a comment - I kind of like Todd's approach. Maybe we should consider adding a sudo command to FsShell so it's not a separate utility. Using an env makes me a bit squeamish since it may introduce an unexpected attack vector.
          Hide
          vicaya Luke Lu added a comment -

          @Owen, I'm fine with repurposing HADOOP_USER_NAME and for proxy user (better auditing and access control even without kerbero), though it's an incompatible change. One of the reasons we added HADOOP_PROXY_USER is to preserve the original semantics for HADOOP_USER_NAME.

          Show
          vicaya Luke Lu added a comment - @Owen, I'm fine with repurposing HADOOP_USER_NAME and for proxy user (better auditing and access control even without kerbero), though it's an incompatible change. One of the reasons we added HADOOP_PROXY_USER is to preserve the original semantics for HADOOP_USER_NAME.
          Hide
          vicaya Luke Lu added a comment -

          We'd also like to use proxy user in "semi" secure mode as well.

          Show
          vicaya Luke Lu added a comment - We'd also like to use proxy user in "semi" secure mode as well.
          Hide
          owen.omalley Owen O'Malley added a comment -

          I'm not against making an environment variable/property to set the user, but we might as well use the one we already have and enable HADOOP_USER_NAME in secure mode to mean act as a proxy for the given user.

          Show
          owen.omalley Owen O'Malley added a comment - I'm not against making an environment variable/property to set the user, but we might as well use the one we already have and enable HADOOP_USER_NAME in secure mode to mean act as a proxy for the given user.
          Show
          tlipcon Todd Lipcon added a comment - We achieved this in Hue with a simple wrapper around FsShell: http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hue/sudo-shell/1.2.0-cdh3u0/com/cloudera/hue/SudoFsShell.java?av=f
          Hide
          revans2 Robert Joseph Evans added a comment -

          This would be really good for testing as well. We have seen issues with HFTP tokens being broken only for proxy users, but were not testing it properly. This should make that testing a lot simpler in the future. +1 for the idea.

          Show
          revans2 Robert Joseph Evans added a comment - This would be really good for testing as well. We have seen issues with HFTP tokens being broken only for proxy users, but were not testing it properly. This should make that testing a lot simpler in the future. +1 for the idea.

            People

            • Assignee:
              crystal_gaoyu Yu Gao
              Reporter:
              vicaya Luke Lu
            • Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development