[HDFS-12139] HTTPFS liststatus returns incorrect pathSuffix for path of file

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-beta1
• Component/s: httpfs
• Hadoop Flags: Reviewed
• Assignee: Yongjun Zhang
• Reporter: Yongjun Zhang
• Created: 14/Jul/17 05:51
• Updated: 02/Oct/19 17:12
• Resolved: 19/Jul/17 19:04

Description

Per the following logs, we can see that liststatus returns the same pathSuffix "test.txt" for /tmp/yj/yj1 and /tmp/yj/yj1/test.txt, which is wrong. The pathSuffix for the latter should be empty.

[thost ~]$ hadoop fs -copyFromLocal test.txt /tmp/yj/yj1
[thost ~]$ curl "http://thost.x.y:14000/webhdfs/v1/tmp/yj/yj1?op=LISTSTATUS&user.name=tuser"
{"FileStatuses":{"FileStatus":[{"pathSuffix":"test.txt","type":"FILE","length":16,"owner":"tuser","group":"supergroup","permission":"644","accessTime":1500007684989,"modificationTime":1500007685286,"blockSize":134217728,"replication":3}]}}
[thost ~]$ curl "http://thost.x.y:14000/webhdfs/v1/tmp/yj/yj1/test.txt?op=LISTSTATUS&user.name=tuser"
{"FileStatuses":{"FileStatus":[{"pathSuffix":"test.txt","type":"FILE","length":16,"owner":"tuser","group":"supergroup","permission":"644","accessTime":1500007684989,"modificationTime":1500007685286,"blockSize":134217728,"replication":3}]}}

Due to this bug, the symptom of running distcp with httpfs to copy a single file /tmp/test.txt is:

WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:systest (auth:PROXY) via httpfs (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=systest, access=EXECUTE, inode="/tmp/test.txt":systest:supergroup:rw-rr- (Ancestor /tmp/test.txt is not a directory).

because /tmp/test.txt is treated as /tmp/test.txt/test.txt.