Integrated in Hadoop-Mapreduce-trunk #1364 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1364/)
HDFS-4542. Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452978)

Result = SUCCESS
kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452978
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java

Integrated in Hadoop-Hdfs-trunk #1336 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1336/)
HDFS-4542. Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452978)

Result = SUCCESS
kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452978
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java

Integrated in Hadoop-Hdfs-0.23-Build #545 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/545/)
HDFS-4542. Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452981)

Result = FAILURE
kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452981
Files :

• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java

Integrated in Hadoop-Yarn-trunk #147 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/147/)
HDFS-4542. Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452978)

Result = SUCCESS
kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452978
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java

Integrated in Hadoop-trunk-Commit #3417 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3417/)
HDFS-4542. Webhdfs doesn't support secure proxy users. Contributed by Daryn Sharp. (Revision 1452978)

Result = SUCCESS
kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1452978
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java

I've committed this to trunk, branch-2 and branch-0.23. Many thanks to Alejandro for reviewing and Daryn for the patch.

Daryn, thanks for the clarifications. +1.

This can't (currently) be relegated to the Authenticator because getHttpUrlConnection does not use an AuthenticatedUrl unless the ugi contains kerberos credentials - which means removing the param will break a non-secure cluster. I didn't change this behavior, so would you file another jira if you really think it's wrong?

Yep, I think this is wrong, filed HDFS-4548.

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12571896/HDFS-4542.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

+1 tests included appear to have a timeout.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4032//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4032//console

This message is automatically generated.

23 patch caused it to skip trunk patch, kick again...

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12571886/HDFS-4542.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

+1 tests included appear to have a timeout.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4030//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4030//console

This message is automatically generated.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12571889/HDFS-4542.branch-23.patch
against trunk revision .

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4031//console

This message is automatically generated.

Patch doesn't apply cleanly to 23 due to diff context. Identical logic to main patch.

Dependency jira is integrated so re-attaching same patch to kick the pre-commit.

Assuming userUgi is the current user the doAs logic is good.

Yes, it is the current user used to instantiate the filesystem.

if (!hasToken), does this means we don't support proxyUser if a DT is being used? Is this defined anyway?

The token itself, from which the connection ugi is created, encodes the proxy user so the server-side works as expected in a proxy context. The DoAsParam can't be sent with tokens because JspHelper will reject a token connection unless UserParam and DoAsParam (if present) match the owner of the token. So the UserParam is expected to be the effective user when used with a token. It's an inconsistency but I think it was designed around a task's ugi.

the authParams.add(new UserParam(userUgi.getShortUserName())) line, we should not do this, it is the AuthenticatedURL via the corresponding Authenticator impl the one responsible for injecting the user.name to the URI

This can't (currently) be relegated to the Authenticator because getHttpUrlConnection does not use an AuthenticatedUrl unless the ugi contains kerberos credentials - which means removing the param will break a non-secure cluster. I didn't change this behavior, so would you file another jira if you really think it's wrong?

Argh, it is OK, was reading wrong the snippet below:

+    UserGroupInformation userUgi = ugi;
+    if (!hasToken) {
+      UserGroupInformation realUgi = userUgi.getRealUser();
+      if (realUgi != null) { // proxy user
+        authParams.add(new DoAsParam(userUgi.getShortUserName()));
+        userUgi = realUgi;
+      }
+    }
+    authParams.add(new UserParam(userUgi.getShortUserName()));
+    return authParams.toArray(new Param<?,?>[0]);

Assuming userUgi is the current user the doAs logic is good.

A couple of extra things in this snippet:

• if (!hasToken), does this means we don't support proxyUser if a DT is being used? Is this defined anyway?
• the authParams.add(new UserParam(userUgi.getShortUserName())) line, we should not do this, it is the AuthenticatedURL via the corresponding Authenticator impl the one responsible for injecting the user.name to the URI.

Sorry Alejandro, I read your comment backwards and essentially repeated what you said. Where do you think I'm doing the wrong thing that the test aren't catching?

Patch is going to fail because of a small tweak to common I neglected to submit first.

Unfortunately, no... The "user" is context sensitive. If there's no "doAs" then the ugi is a plain non-proxy user. If both "user" and "doAs" are provided, then "user" is the real/privileged user, and "doAs" is the effective user.

I really wish "user" always meant effective user, and there was an optional "realUser" for the privileged user, but that would be an incompatible change.

the doAs user should be the current user, not the real user, no? the real user is the user with proxyuser privilege

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12571649/HDFS-4542.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

+1 tests included appear to have a timeout.

-1 javac. The patch appears to cause the build to fail.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4027//console

This message is automatically generated.

Will soon post back-port to 23.