Hadoop Common
  1. Hadoop Common
  2. HADOOP-8561

Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.3-alpha, 0.23.6, 1.1.2
    • Component/s: security
    • Labels:
      None
    • Target Version/s:

      Description

      To solve the problem for an authenticated user to type hadoop shell commands in a web console, we can introduce an HADOOP_PROXY_USER environment variable to allow proper impersonation in the child hadoop client processes.

      1. hadoop-8561-v2.patch
        4 kB
        Luke Lu
      2. hadoop-8561-branch-2.patch
        4 kB
        Yu Gao
      3. hadoop-8561-branch-1.patch
        4 kB
        Yu Gao
      4. hadoop-8561.patch
        4 kB
        Yu Gao

        Issue Links

          Activity

          Luke Lu created issue -
          Hide
          Robert Joseph Evans added a comment -

          This would be really good for testing as well. We have seen issues with HFTP tokens being broken only for proxy users, but were not testing it properly. This should make that testing a lot simpler in the future. +1 for the idea.

          Show
          Robert Joseph Evans added a comment - This would be really good for testing as well. We have seen issues with HFTP tokens being broken only for proxy users, but were not testing it properly. This should make that testing a lot simpler in the future. +1 for the idea.
          Show
          Todd Lipcon added a comment - We achieved this in Hue with a simple wrapper around FsShell: http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hue/sudo-shell/1.2.0-cdh3u0/com/cloudera/hue/SudoFsShell.java?av=f
          Hide
          Owen O'Malley added a comment -

          I'm not against making an environment variable/property to set the user, but we might as well use the one we already have and enable HADOOP_USER_NAME in secure mode to mean act as a proxy for the given user.

          Show
          Owen O'Malley added a comment - I'm not against making an environment variable/property to set the user, but we might as well use the one we already have and enable HADOOP_USER_NAME in secure mode to mean act as a proxy for the given user.
          Hide
          Luke Lu added a comment -

          We'd also like to use proxy user in "semi" secure mode as well.

          Show
          Luke Lu added a comment - We'd also like to use proxy user in "semi" secure mode as well.
          Hide
          Luke Lu added a comment -

          @Owen, I'm fine with repurposing HADOOP_USER_NAME and for proxy user (better auditing and access control even without kerbero), though it's an incompatible change. One of the reasons we added HADOOP_PROXY_USER is to preserve the original semantics for HADOOP_USER_NAME.

          Show
          Luke Lu added a comment - @Owen, I'm fine with repurposing HADOOP_USER_NAME and for proxy user (better auditing and access control even without kerbero), though it's an incompatible change. One of the reasons we added HADOOP_PROXY_USER is to preserve the original semantics for HADOOP_USER_NAME.
          Hide
          Daryn Sharp added a comment -

          I kind of like Todd's approach. Maybe we should consider adding a sudo command to FsShell so it's not a separate utility. Using an env makes me a bit squeamish since it may introduce an unexpected attack vector.

          Show
          Daryn Sharp added a comment - I kind of like Todd's approach. Maybe we should consider adding a sudo command to FsShell so it's not a separate utility. Using an env makes me a bit squeamish since it may introduce an unexpected attack vector.
          Matt Foley made changes -
          Field Original Value New Value
          Target Version/s 1.1.0 [ 12316501 ] 1.2.0 [ 12321659 ]
          Hide
          Matt Foley added a comment -

          Moved to 1.2.0 upon release of 1.1.0.

          Show
          Matt Foley added a comment - Moved to 1.2.0 upon release of 1.1.0.
          Yu Gao made changes -
          Attachment hadoop-8561-branch-1.patch [ 12551909 ]
          Yu Gao made changes -
          Attachment hadoop-8561-branch-2.patch [ 12551910 ]
          Yu Gao made changes -
          Attachment hadoop-8561.patch [ 12551912 ]
          Hide
          Luke Lu added a comment -

          This approach has added benefit of working with clients (like HBase shell) not written in Java.

          Using an env makes me a bit squeamish since it may introduce an unexpected attack vector.

          It won't do anything for ordinary users. An admin web app of course needs to do a few things sanitize the input to disallow fork/exec etc.

          Show
          Luke Lu added a comment - This approach has added benefit of working with clients (like HBase shell) not written in Java. Using an env makes me a bit squeamish since it may introduce an unexpected attack vector. It won't do anything for ordinary users. An admin web app of course needs to do a few things sanitize the input to disallow fork/exec etc.
          Hide
          Luke Lu added a comment -

          The patches lgtm. +1 pending jenkins.

          Show
          Luke Lu added a comment - The patches lgtm. +1 pending jenkins.
          Luke Lu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags Reviewed [ 10343 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551912/hadoop-8561.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1701//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1701//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551912/hadoop-8561.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1701//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1701//console This message is automatically generated.
          Hide
          Luke Lu added a comment -

          Need to merge with HADOOP-9035

          Show
          Luke Lu added a comment - Need to merge with HADOOP-9035
          Luke Lu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Luke Lu made changes -
          Attachment hadoop-8561-v2.patch [ 12561097 ]
          Luke Lu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags Reviewed [ 10343 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12561097/hadoop-8561-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1878//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1878//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12561097/hadoop-8561-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1878//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1878//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3128 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3128/)
          HADOOP-8561. Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429)

          Result = SUCCESS
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3128 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3128/ ) HADOOP-8561 . Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Hide
          Luke Lu added a comment -

          Committed to trunk and branch-

          {2,1,1.1}

          . Thanks Yu!

          Show
          Luke Lu added a comment - Committed to trunk and branch- {2,1,1.1} . Thanks Yu!
          Luke Lu made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 1.2.0 [ 12321659 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Fix Version/s 2.0.3-alpha [ 12323273 ]
          Fix Version/s 1.1.2 [ 12323596 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #67 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/67/)
          HADOOP-8561. Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429)

          Result = SUCCESS
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #67 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/67/ ) HADOOP-8561 . Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1256 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1256/)
          HADOOP-8561. Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429)

          Result = FAILURE
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1256 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1256/ ) HADOOP-8561 . Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429) Result = FAILURE llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1287 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1287/)
          HADOOP-8561. Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429)

          Result = SUCCESS
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1287 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1287/ ) HADOOP-8561 . Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes. (Yu Gao via llu) (Revision 1422429) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1422429 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Thomas Graves made changes -
          Fix Version/s 0.23.6 [ 12323504 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #470 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/470/)
          HADOOP-8561. Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes (Yu Gao via tgraves) (Revision 1424698)

          Result = UNSTABLE
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424698
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #470 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/470/ ) HADOOP-8561 . Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes (Yu Gao via tgraves) (Revision 1424698) Result = UNSTABLE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424698 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestProxyUserFromEnv.java
          Matt Foley made changes -
          Fix Version/s 1.2.0 [ 12321659 ]
          Matt Foley made changes -
          Target Version/s 1.2.0 [ 12321659 ] 1.1.2 [ 12323596 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Allen Wittenauer made changes -
          Link This issue duplicates HADOOP-6457 [ HADOOP-6457 ]
          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320357 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          42d 9h 24m 1 Luke Lu 15/Dec/12 04:44
          Open Open Patch Available Patch Available
          120d 9h 12m 2 Luke Lu 15/Dec/12 04:45
          Patch Available Patch Available Resolved Resolved
          22h 8m 1 Luke Lu 16/Dec/12 02:54
          Resolved Resolved Closed Closed
          61d 10h 17m 1 Arun C Murthy 15/Feb/13 13:12

            People

            • Assignee:
              Yu Gao
              Reporter:
              Luke Lu
            • Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development