Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.6.0
-
Fix Version/s: 2.8.0, 3.0.0-alpha1
-
Component/s: webhdfs
-
Labels:None
-
Hadoop Flags:Reviewed
Description
Per the spec, WebHDFS provides a REST endpoint for getting the user's home directory:
Submit a HTTP GET request. curl -i "http://<HOST>:<PORT>/webhdfs/v1/?op=GETHOMEDIRECTORY"
However, WebHDFSFileSystem.java does not use this, instead building the home directory locally:
/** @return the home directory. */ public static String getHomeDirectoryString(final UserGroupInformation ugi) { return "/user/" + ugi.getShortUserName(); } @Override public Path getHomeDirectory() { return makeQualified(new Path(getHomeDirectoryString(ugi))); }
The WebHDFSFileSystem client should call to the REST service to determine the home directory.
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Attached patch to consider the user from request and direct the call to actual namenode which intern returns the requested user's home directory rather than hard coded paths.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 18m 41s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | javac | 7m 45s | There were no new javac warning messages. |
| +1 | javadoc | 9m 52s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. |
| +1 | checkstyle | 2m 40s | There were no new checkstyle issues. |
| +1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 34s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 4m 8s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | native | 3m 19s | Pre-build of native portion |
| -1 | hdfs tests | 171m 23s | Tests failed in hadoop-hdfs. |
| +1 | hdfs tests | 0m 17s | Tests passed in hadoop-hdfs-client. |
| 220m 39s |
| Reason | Tests |
|---|---|
| Failed unit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| hadoop.hdfs.security.TestDelegationTokenForProxyUser | |
| hadoop.hdfs.web.TestWebHdfsFileSystemContract | |
| hadoop.hdfs.web.TestWebHdfsTokens | |
| hadoop.hdfs.web.TestWebHdfsContentLength |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12739053/HDFS-8542-00.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 95c73d4 |
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11312/artifact/patchprocess/testrun_hadoop-hdfs.txt |
| hadoop-hdfs-client test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11312/artifact/patchprocess/testrun_hadoop-hdfs-client.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/11312/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11312/console |
This message was automatically generated.
Update patch for the test failures
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 19m 9s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | javac | 7m 39s | There were no new javac warning messages. |
| +1 | javadoc | 9m 40s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. |
| -1 | checkstyle | 2m 40s | The applied patch generated 1 new checkstyle issues (total was 132, now 133). |
| +1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 36s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 4m 7s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | native | 3m 14s | Pre-build of native portion |
| +1 | hdfs tests | 162m 33s | Tests passed in hadoop-hdfs. |
| +1 | hdfs tests | 0m 17s | Tests passed in hadoop-hdfs-client. |
| 211m 52s |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12739419/HDFS-8542-01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eef7b50 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/11338/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt |
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11338/artifact/patchprocess/testrun_hadoop-hdfs.txt |
| hadoop-hdfs-client test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11338/artifact/patchprocess/testrun_hadoop-hdfs-client.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/11338/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11338/console |
This message was automatically generated.
check style issue reported doesn't seem to be valid because in my eclipse the indention looks fine and matches with other similar code snippets in the file
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java:287: '.' have incorrect indentation level 9, expected level should be 12.
- The checkstyle ding is a bit convoluted, but probably correct since the matching brace is nested pretty heavily. Let's make checkstyle happy and give it what it wants.
- Please remove the println from the unit test
- Not sure it's worth caching the response here. Home directories shouldn't change, but there's no huge reason why they couldn't between calls. Since there's no way to clear the cache and this call is pretty light, I'd rather not cache. Not caching will simplify the code.
- The existing org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java is incorrectly testing the getHomeDirectory method:
{//test GETHOMEDIRECTORY final URL url = webhdfs.toUrl(GetOpParam.Op.GETHOMEDIRECTORY, root); final HttpURLConnection conn = (HttpURLConnection) url.openConnection(); final Map<?, ?> m = WebHdfsTestUtil.connectAndGetJson( conn, HttpServletResponse.SC_OK); assertEquals(WebHdfsFileSystem.getHomeDirectoryString(ugi), m.get(Path.class.getSimpleName())); conn.disconnect(); }since it's calling the static getHomeDirectoryString method rather than the instance getHomeDirectory method. We should fix this and deprecate the static getHomeDirectoryString method since there's no use (or callers) for it.
Otherwise looks good. Thanks.
Canceling patch post-review.
Thank you Jakob Homan for the review. I have uploaded new patch by fixing all the comments mentioned except the one about caching.
I have considered caching to address based on the discussion in HDFS-3620 (Suggested by Alejandro Abdelnur). In my opinion also caching is better to avoid multiple HTTP calls.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| -1 | pre-patch | 16m 22s | Findbugs (version 3.0.0) appears to be broken on trunk. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 2 new or modified test files. |
| +1 | javac | 7m 41s | There were no new javac warning messages. |
| +1 | javadoc | 9m 52s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
| +1 | checkstyle | 1m 7s | There were no new checkstyle issues. |
| +1 | whitespace | 0m 1s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 38s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 4m 8s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | native | 3m 21s | Pre-build of native portion |
| -1 | hdfs tests | 159m 54s | Tests failed in hadoop-hdfs. |
| +1 | hdfs tests | 0m 16s | Tests passed in hadoop-hdfs-client. |
| 205m 21s |
| Reason | Tests |
|---|---|
| Failed unit tests | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| hadoop.hdfs.TestEncryptionZonesWithKMS |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12741037/HDFS-8542-02.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 445b132 |
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11435/artifact/patchprocess/testrun_hadoop-hdfs.txt |
| hadoop-hdfs-client test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11435/artifact/patchprocess/testrun_hadoop-hdfs-client.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/11435/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11435/console |
This message was automatically generated.
I'm still not wild about caching the result since again, (a) the value is never discarded, so it's not a cache and (b) backing systems could choose to change this value on a subsequent call. However, both FileSystem and DistributedFileSystem are doing some questionable things with this API, so I'll worry about those issues later, if we run into them.
+1 on current patch. Failed tests are spurious. Attaching a version for 2.7 (same except location of JsonUtils). Will commit both after Jenkins has a pass over backport.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| -1 | patch | 0m 0s | The patch command could not apply the patch during dryrun. |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12741136/HDFS-8542-branch-2.7.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 7b424f9 |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11439/console |
This message was automatically generated.
Jenkins isn't running against minor versions. I've committed this to trunk and branch-2. Thanks, Kanaka. Resolving.
FAILURE: Integrated in Hadoop-trunk-Commit #8049 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8049/)
HDFS-8542. WebHDFS getHomeDirectory behavior does not match specification. Contributed by Kanaka Kumar Avvaru. (jghoman: rev fac4e04dd359a7ff31f286d664fb06f019ec0b58)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Thanks Jakob Homan for the reviews and commit.
FAILURE: Integrated in Hadoop-Yarn-trunk #967 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/967/)
HDFS-8542. WebHDFS getHomeDirectory behavior does not match specification. Contributed by Kanaka Kumar Avvaru. (jghoman: rev fac4e04dd359a7ff31f286d664fb06f019ec0b58)
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #237 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/237/)
HDFS-8542. WebHDFS getHomeDirectory behavior does not match specification. Contributed by Kanaka Kumar Avvaru. (jghoman: rev fac4e04dd359a7ff31f286d664fb06f019ec0b58)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
FAILURE: Integrated in Hadoop-Hdfs-trunk #2165 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2165/)
HDFS-8542. WebHDFS getHomeDirectory behavior does not match specification. Contributed by Kanaka Kumar Avvaru. (jghoman: rev fac4e04dd359a7ff31f286d664fb06f019ec0b58)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #226 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/226/)
HDFS-8542. WebHDFS getHomeDirectory behavior does not match specification. Contributed by Kanaka Kumar Avvaru. (jghoman: rev fac4e04dd359a7ff31f286d664fb06f019ec0b58)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #235 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/235/)
HDFS-8542. WebHDFS getHomeDirectory behavior does not match specification. Contributed by Kanaka Kumar Avvaru. (jghoman: rev fac4e04dd359a7ff31f286d664fb06f019ec0b58)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2183/)
HDFS-8542. WebHDFS getHomeDirectory behavior does not match specification. Contributed by Kanaka Kumar Avvaru. (jghoman: rev fac4e04dd359a7ff31f286d664fb06f019ec0b58)
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
Hi Jakob Homan thanks for reporting. I would like to work on this JIRA as it's status is un-assigned. If you have already started working, please feel free to assign to you.