Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 2.0.0-alpha, 3.0.0
    • Fix Version/s: 0.23.7, 2.1.0-beta
    • Component/s: webhdfs
    • Labels:
      None

      Description

      Webhdfs doesn't ever send the DoAsParam in the REST calls for proxy users. Proxy users on a non-secure cluster "work" because the server sees them as the effective user, not a proxy user, which effectively bypasses the proxy authorization checks. On secure clusters, it doesn't work at all in part due to wrong ugi being used for the connection (HDFS-3367), but then it fails because the effective user tries to use a non-proxy token for the real user.

      1. HDFS-4542.patch
        19 kB
        Daryn Sharp
      2. HDFS-4542.branch-23.patch
        19 kB
        Daryn Sharp
      3. HDFS-4542.patch
        19 kB
        Daryn Sharp

        Issue Links

          Activity

          Hide
          Daryn Sharp added a comment -

          Will soon post back-port to 23.

          Show
          Daryn Sharp added a comment - Will soon post back-port to 23.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12571649/HDFS-4542.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 tests included appear to have a timeout.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4027//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571649/HDFS-4542.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4027//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          the doAs user should be the current user, not the real user, no? the real user is the user with proxyuser privilege

          Show
          Alejandro Abdelnur added a comment - the doAs user should be the current user, not the real user, no? the real user is the user with proxyuser privilege
          Hide
          Daryn Sharp added a comment -

          Unfortunately, no... The "user" is context sensitive. If there's no "doAs" then the ugi is a plain non-proxy user. If both "user" and "doAs" are provided, then "user" is the real/privileged user, and "doAs" is the effective user.

          I really wish "user" always meant effective user, and there was an optional "realUser" for the privileged user, but that would be an incompatible change.

          Show
          Daryn Sharp added a comment - Unfortunately, no... The "user" is context sensitive. If there's no "doAs" then the ugi is a plain non-proxy user. If both "user" and "doAs" are provided, then "user" is the real/privileged user, and "doAs" is the effective user. I really wish "user" always meant effective user, and there was an optional "realUser" for the privileged user, but that would be an incompatible change.
          Hide
          Daryn Sharp added a comment -

          Patch is going to fail because of a small tweak to common I neglected to submit first.

          Show
          Daryn Sharp added a comment - Patch is going to fail because of a small tweak to common I neglected to submit first.
          Hide
          Daryn Sharp added a comment -

          Sorry Alejandro, I read your comment backwards and essentially repeated what you said. Where do you think I'm doing the wrong thing that the test aren't catching?

          Show
          Daryn Sharp added a comment - Sorry Alejandro, I read your comment backwards and essentially repeated what you said. Where do you think I'm doing the wrong thing that the test aren't catching?
          Hide
          Alejandro Abdelnur added a comment -

          Argh, it is OK, was reading wrong the snippet below:

          +    UserGroupInformation userUgi = ugi;
          +    if (!hasToken) {
          +      UserGroupInformation realUgi = userUgi.getRealUser();
          +      if (realUgi != null) { // proxy user
          +        authParams.add(new DoAsParam(userUgi.getShortUserName()));
          +        userUgi = realUgi;
          +      }
          +    }
          +    authParams.add(new UserParam(userUgi.getShortUserName()));
          +    return authParams.toArray(new Param<?,?>[0]);
          

          Assuming userUgi is the current user the doAs logic is good.

          A couple of extra things in this snippet:

          • if (!hasToken), does this means we don't support proxyUser if a DT is being used? Is this defined anyway?
          • the authParams.add(new UserParam(userUgi.getShortUserName())) line, we should not do this, it is the AuthenticatedURL via the corresponding Authenticator impl the one responsible for injecting the user.name to the URI.
          Show
          Alejandro Abdelnur added a comment - Argh, it is OK, was reading wrong the snippet below: + UserGroupInformation userUgi = ugi; + if (!hasToken) { + UserGroupInformation realUgi = userUgi.getRealUser(); + if (realUgi != null ) { // proxy user + authParams.add( new DoAsParam(userUgi.getShortUserName())); + userUgi = realUgi; + } + } + authParams.add( new UserParam(userUgi.getShortUserName())); + return authParams.toArray( new Param<?,?>[0]); Assuming userUgi is the current user the doAs logic is good. A couple of extra things in this snippet: if (!hasToken) , does this means we don't support proxyUser if a DT is being used? Is this defined anyway? the authParams.add(new UserParam(userUgi.getShortUserName())) line, we should not do this, it is the AuthenticatedURL via the corresponding Authenticator impl the one responsible for injecting the user.name to the URI.
          Hide
          Daryn Sharp added a comment -

          Assuming userUgi is the current user the doAs logic is good.

          Yes, it is the current user used to instantiate the filesystem.

          if (!hasToken), does this means we don't support proxyUser if a DT is being used? Is this defined anyway?

          The token itself, from which the connection ugi is created, encodes the proxy user so the server-side works as expected in a proxy context. The DoAsParam can't be sent with tokens because JspHelper will reject a token connection unless UserParam and DoAsParam (if present) match the owner of the token. So the UserParam is expected to be the effective user when used with a token. It's an inconsistency but I think it was designed around a task's ugi.

          the authParams.add(new UserParam(userUgi.getShortUserName())) line, we should not do this, it is the AuthenticatedURL via the corresponding Authenticator impl the one responsible for injecting the user.name to the URI

          This can't (currently) be relegated to the Authenticator because getHttpUrlConnection does not use an AuthenticatedUrl unless the ugi contains kerberos credentials - which means removing the param will break a non-secure cluster. I didn't change this behavior, so would you file another jira if you really think it's wrong?

          Show
          Daryn Sharp added a comment - Assuming userUgi is the current user the doAs logic is good. Yes, it is the current user used to instantiate the filesystem. if (!hasToken), does this means we don't support proxyUser if a DT is being used? Is this defined anyway? The token itself, from which the connection ugi is created, encodes the proxy user so the server-side works as expected in a proxy context. The DoAsParam can't be sent with tokens because JspHelper will reject a token connection unless UserParam and DoAsParam (if present) match the owner of the token. So the UserParam is expected to be the effective user when used with a token. It's an inconsistency but I think it was designed around a task's ugi. the authParams.add(new UserParam(userUgi.getShortUserName())) line, we should not do this, it is the AuthenticatedURL via the corresponding Authenticator impl the one responsible for injecting the user.name to the URI This can't (currently) be relegated to the Authenticator because getHttpUrlConnection does not use an AuthenticatedUrl unless the ugi contains kerberos credentials - which means removing the param will break a non-secure cluster. I didn't change this behavior, so would you file another jira if you really think it's wrong?
          Hide
          Daryn Sharp added a comment -

          Dependency jira is integrated so re-attaching same patch to kick the pre-commit.

          Show
          Daryn Sharp added a comment - Dependency jira is integrated so re-attaching same patch to kick the pre-commit.
          Hide
          Daryn Sharp added a comment -

          Patch doesn't apply cleanly to 23 due to diff context. Identical logic to main patch.

          Show
          Daryn Sharp added a comment - Patch doesn't apply cleanly to 23 due to diff context. Identical logic to main patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12571889/HDFS-4542.branch-23.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4031//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571889/HDFS-4542.branch-23.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4031//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12571886/HDFS-4542.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4030//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4030//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571886/HDFS-4542.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4030//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4030//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          23 patch caused it to skip trunk patch, kick again...

          Show
          Daryn Sharp added a comment - 23 patch caused it to skip trunk patch, kick again...
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12571896/HDFS-4542.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4032//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4032//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571896/HDFS-4542.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4032//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4032//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          Daryn, thanks for the clarifications. +1.

          This can't (currently) be relegated to the Authenticator because getHttpUrlConnection does not use an AuthenticatedUrl unless the ugi contains kerberos credentials - which means removing the param will break a non-secure cluster. I didn't change this behavior, so would you file another jira if you really think it's wrong?

          Yep, I think this is wrong, filed HDFS-4548.

          Show
          Alejandro Abdelnur added a comment - Daryn, thanks for the clarifications. +1. This can't (currently) be relegated to the Authenticator because getHttpUrlConnection does not use an AuthenticatedUrl unless the ugi contains kerberos credentials - which means removing the param will break a non-secure cluster. I didn't change this behavior, so would you file another jira if you really think it's wrong? Yep, I think this is wrong, filed HDFS-4548 .
          Hide
          Kihwal Lee added a comment -

          I've committed this to trunk, branch-2 and branch-0.23. Many thanks to Alejandro for reviewing and Daryn for the patch.

          Show
          Kihwal Lee added a comment - I've committed this to trunk, branch-2 and branch-0.23. Many thanks to Alejandro for reviewing and Daryn for the patch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3417 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3417/)
          HDFS-4542. Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452978)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452978
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3417 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3417/ ) HDFS-4542 . Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452978) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452978 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #147 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/147/)
          HDFS-4542. Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452978)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452978
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #147 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/147/ ) HDFS-4542 . Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452978) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452978 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #545 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/545/)
          HDFS-4542. Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452981)

          Result = FAILURE
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452981
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #545 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/545/ ) HDFS-4542 . Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452981) Result = FAILURE kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452981 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1336 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1336/)
          HDFS-4542. Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452978)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452978
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1336 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1336/ ) HDFS-4542 . Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452978) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452978 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1364 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1364/)
          HDFS-4542. Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452978)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452978
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1364 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1364/ ) HDFS-4542 . Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452978) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452978 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development