Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12139

HTTPFS liststatus returns incorrect pathSuffix for path of file

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: httpfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Per the following logs, we can see that liststatus returns the same pathSuffix "test.txt" for /tmp/yj/yj1 and /tmp/yj/yj1/test.txt, which is wrong. The pathSuffix for the latter should be empty.

      [thost ~]$ hadoop fs -copyFromLocal test.txt /tmp/yj/yj1
      [thost ~]$ curl "http://thost.x.y:14000/webhdfs/v1/tmp/yj/yj1?op=LISTSTATUS&user.name=tuser"
      {"FileStatuses":{"FileStatus":[{"pathSuffix":"test.txt","type":"FILE","length":16,"owner":"tuser","group":"supergroup","permission":"644","accessTime":1500007684989,"modificationTime":1500007685286,"blockSize":134217728,"replication":3}]}}
      [thost ~]$ curl "http://thost.x.y:14000/webhdfs/v1/tmp/yj/yj1/test.txt?op=LISTSTATUS&user.name=tuser"
      {"FileStatuses":{"FileStatus":[{"pathSuffix":"test.txt","type":"FILE","length":16,"owner":"tuser","group":"supergroup","permission":"644","accessTime":1500007684989,"modificationTime":1500007685286,"blockSize":134217728,"replication":3}]}}
      

      Due to this bug, the symptom of running distcp with httpfs to copy a single file /tmp/test.txt is:

      WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:systest (auth:PROXY) via httpfs (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=systest, access=EXECUTE, inode="/tmp/test.txt":systest:supergroup:rw-rr- (Ancestor /tmp/test.txt is not a directory).

      because /tmp/test.txt is treated as /tmp/test.txt/test.txt.

        Attachments

        1. HDFS-12139.001.patch
          3 kB
          Yongjun Zhang
        2. HDFS-12139.002.patch
          5 kB
          Yongjun Zhang

          Activity

            People

            • Assignee:
              yzhangal Yongjun Zhang
              Reporter:
              yzhangal Yongjun Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: