[HDFS-14369] RBF: Fix trailing "/" for webhdfs

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0, HDFS-13891
• Hadoop Flags: Reviewed
• Assignee: Akira Ajisaka
• Reporter: CR Hota
• Created: 13/Mar/19 19:36
• Updated: 12/Sep/19 20:03
• Resolved: 08/Apr/19 16:17

Description

WebHDFS doesn't trim trailing slash causing discrepancy in operations.

Example below
--------------------------
Using HDFS API, two directory are listed.

$ hdfs dfs -ls hdfs://<router>:8888/tmp/
Found 2 items
drwxrwxrwx   - hdfs supergroup          0 2018-11-09 17:50 hdfs://<router>:8888/tmp/tmp1
drwxrwxrwx   - hdfs supergroup          0 2018-11-09 17:50 hdfs://<router>:8888/tmp/tmp2

Using WebHDFS API, only one directory is listed.

$ curl -u : --negotiate -i "http://<router>:50071/webhdfs/v1/tmp/?op=LISTSTATUS"
(snip)
{"FileStatuses":{"FileStatus":[
{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1552016766769,"owner":"hdfs","pathSuffix":"tmp1","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
]}}

The mount table is as follows:

$ hdfs dfsrouteradmin -ls /tmp
Mount Table Entries:
Source                    Destinations              Owner                     Group                     Mode                      Quota/Usage              
/tmp                      ns1->/tmp                 aajisaka                  users                     rwxr-xr-x                 [NsQuota: -/-, SsQuota: -/-]
/tmp/tmp1                 ns1->/tmp/tmp1            aajisaka                  users                     rwxr-xr-x                 [NsQuota: -/-, SsQuota: -/-]
/tmp/tmp2                 ns2->/tmp/tmp2            aajisaka                  users                     rwxr-xr-x                 [NsQuota: -/-, SsQuota: -/-]

Without trailing thrash, two directories are listed.

$ curl -u : --negotiate -i "http://<router>:50071/webhdfs/v1/tmp?op=LISTSTATUS"
(snip)
{"FileStatuses":{"FileStatus":[
{"accessTime":1541753421917,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753421917,"owner":"hdfs","pathSuffix":"tmp1","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"},
{"accessTime":1541753429812,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753429812,"owner":"hdfs","pathSuffix":"tmp2","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"}
]}}

ajisakaa Thanks for reporting this, I borrowed the text from
HDFS-13972

Issue Links

• depends upon: HADOOP-16226 new Path(String str) does not remove all the trailing slashes of str