Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12139

HTTPFS liststatus returns incorrect pathSuffix for path of file

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: httpfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Per the following logs, we can see that liststatus returns the same pathSuffix "test.txt" for /tmp/yj/yj1 and /tmp/yj/yj1/test.txt, which is wrong. The pathSuffix for the latter should be empty.

      [thost ~]$ hadoop fs -copyFromLocal test.txt /tmp/yj/yj1
      [thost ~]$ curl "http://thost.x.y:14000/webhdfs/v1/tmp/yj/yj1?op=LISTSTATUS&user.name=tuser"
      {"FileStatuses":{"FileStatus":[{"pathSuffix":"test.txt","type":"FILE","length":16,"owner":"tuser","group":"supergroup","permission":"644","accessTime":1500007684989,"modificationTime":1500007685286,"blockSize":134217728,"replication":3}]}}
      [thost ~]$ curl "http://thost.x.y:14000/webhdfs/v1/tmp/yj/yj1/test.txt?op=LISTSTATUS&user.name=tuser"
      {"FileStatuses":{"FileStatus":[{"pathSuffix":"test.txt","type":"FILE","length":16,"owner":"tuser","group":"supergroup","permission":"644","accessTime":1500007684989,"modificationTime":1500007685286,"blockSize":134217728,"replication":3}]}}
      

      Due to this bug, the symptom of running distcp with httpfs to copy a single file /tmp/test.txt is:

      WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:systest (auth:PROXY) via httpfs (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=systest, access=EXECUTE, inode="/tmp/test.txt":systest:supergroup:rw-rr- (Ancestor /tmp/test.txt is not a directory).

      because /tmp/test.txt is treated as /tmp/test.txt/test.txt.

      1. HDFS-12139.001.patch
        3 kB
        Yongjun Zhang
      2. HDFS-12139.002.patch
        5 kB
        Yongjun Zhang

        Activity

        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks a lot Xiao Chen, committed to trunk and branch-2. I also created HDFS-12162 per our discussion.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks a lot Xiao Chen , committed to trunk and branch-2. I also created HDFS-12162 per our discussion.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12034 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12034/)
        HDFS-12139. HTTPFS liststatus returns incorrect pathSuffix for path of (yzhang: rev 3556e36be30211f46ac38899ce11a4d4cac6d635)

        • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12034 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12034/ ) HDFS-12139 . HTTPFS liststatus returns incorrect pathSuffix for path of (yzhang: rev 3556e36be30211f46ac38899ce11a4d4cac6d635) (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
        Hide
        xiaochen Xiao Chen added a comment -

        +1. Thanks, Yongjun Zhang.

        Show
        xiaochen Xiao Chen added a comment - +1. Thanks, Yongjun Zhang .
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 3m 20s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 11s trunk passed
        +1 compile 0m 21s trunk passed
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 2m 20s trunk passed
        +1 findbugs 0m 27s trunk passed
        +1 javadoc 0m 14s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 19s the patch passed
        +1 compile 0m 17s the patch passed
        +1 javac 0m 17s the patch passed
        +1 checkstyle 0m 12s the patch passed
        +1 mvnsite 2m 22s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 31s the patch passed
        +1 javadoc 0m 12s the patch passed
              Other Tests
        +1 unit 3m 24s hadoop-hdfs-httpfs in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        28m 58s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12139
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877895/HDFS-12139.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7fa707da2c02 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5aa2bf2
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20330/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20330/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 3m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 11s trunk passed +1 compile 0m 21s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 2m 20s trunk passed +1 findbugs 0m 27s trunk passed +1 javadoc 0m 14s trunk passed       Patch Compile Tests +1 mvninstall 0m 19s the patch passed +1 compile 0m 17s the patch passed +1 javac 0m 17s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 2m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 31s the patch passed +1 javadoc 0m 12s the patch passed       Other Tests +1 unit 3m 24s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 28m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12139 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877895/HDFS-12139.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7fa707da2c02 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5aa2bf2 Default Java 1.8.0_131 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20330/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20330/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yzhangal Yongjun Zhang added a comment -

        All good catches, thanks much Xiao Chen! I just uploaded rev2 to address most of them.

        About the assertEquals, most of the statements there are cross-checking each other, and I tried to use a convenient order, for that ones that do have an exact expected value, I reordered the parameters. Appreciate your look at rev2.

        Show
        yzhangal Yongjun Zhang added a comment - All good catches, thanks much Xiao Chen ! I just uploaded rev2 to address most of them. About the assertEquals, most of the statements there are cross-checking each other, and I tried to use a convenient order, for that ones that do have an exact expected value, I reordered the parameters. Appreciate your look at rev2.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Yongjun Zhang for clarifying offline. Change makes sense to me.

        Some comments below:

        • It seems we should also fix the same for LISTSTATUS_BATCH. I observed failures with the following snippet:
              RemoteIterator<FileStatus> ri = fs.listStatusIterator(path);
              assertTrue(ri.hasNext());
              FileStatus fff = ri.next();
              Assert.assertEquals(fff.getPath().getName(), path.getName());
              Assert.assertEquals(fff.getPath(), statl[0].getPath());
              Assert.assertEquals(fff.getPath(), statl[0].getPath());
          
        • Suggest assertEquals to follow the signature of (expected, actual). (i.e. s/assertEquals(statl.length, 1);/assertEquals(1, statl.length);/g
        • isFile is deprecated, how about we change to check it from isDirectory from the file status?
        • Javadoc typo: s/fie path/file path/g
        Show
        xiaochen Xiao Chen added a comment - Thanks Yongjun Zhang for clarifying offline. Change makes sense to me. Some comments below: It seems we should also fix the same for LISTSTATUS_BATCH . I observed failures with the following snippet: RemoteIterator<FileStatus> ri = fs.listStatusIterator(path); assertTrue(ri.hasNext()); FileStatus fff = ri.next(); Assert.assertEquals(fff.getPath().getName(), path.getName()); Assert.assertEquals(fff.getPath(), statl[0].getPath()); Assert.assertEquals(fff.getPath(), statl[0].getPath()); Suggest assertEquals to follow the signature of (expected, actual). (i.e. s/assertEquals(statl.length, 1);/assertEquals(1, statl.length);/g isFile is deprecated, how about we change to check it from isDirectory from the file status? Javadoc typo: s/fie path/file path/g
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Rushabh S Shah Xiao Chen Wei-Chiu Chuang.

        This seems expected, according to http://hadoop.apache.org/docs/r3.0.0-alpha2/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#List_a_Directory ?

        The issue here is about httpfs liststatus on path to a file.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Rushabh S Shah Xiao Chen Wei-Chiu Chuang . This seems expected, according to http://hadoop.apache.org/docs/r3.0.0-alpha2/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#List_a_Directory ? The issue here is about httpfs liststatus on path to a file.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Well not expected, since WebHDFS does not have the same problem. This is a httpfs specific bug.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Well not expected, since WebHDFS does not have the same problem. This is a httpfs specific bug.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Yongjun Zhang for creating the issue and providing a patch. Updated title as suggested by Rushabh.

        This seems expected, according to http://hadoop.apache.org/docs/r3.0.0-alpha2/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#List_a_Directory ?

        Show
        xiaochen Xiao Chen added a comment - Thanks Yongjun Zhang for creating the issue and providing a patch. Updated title as suggested by Rushabh. This seems expected, according to http://hadoop.apache.org/docs/r3.0.0-alpha2/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#List_a_Directory ?
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Hi Yongjun Zhang, I haven't gone through patch but is this bug just in HTTP file system ?
        If yes then can you please change the title to reflect that ?

        Show
        shahrs87 Rushabh S Shah added a comment - Hi Yongjun Zhang , I haven't gone through patch but is this bug just in HTTP file system ? If yes then can you please change the title to reflect that ?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 1m 52s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 12s trunk passed
        +1 compile 0m 18s trunk passed
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 2m 21s trunk passed
        +1 findbugs 0m 25s trunk passed
        +1 javadoc 0m 15s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 16s the patch passed
        +1 compile 0m 15s the patch passed
        -1 javac 0m 15s hadoop-hdfs-project_hadoop-hdfs-httpfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        +1 checkstyle 0m 11s the patch passed
        +1 mvnsite 2m 17s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 30s the patch passed
        +1 javadoc 0m 11s the patch passed
              Other Tests
        +1 unit 3m 24s hadoop-hdfs-httpfs in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        27m 14s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12139
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877526/HDFS-12139.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e5bf79f886f7 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 02b141a
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/20302/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20302/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20302/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 1m 52s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 12s trunk passed +1 compile 0m 18s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 2m 21s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 15s trunk passed       Patch Compile Tests +1 mvninstall 0m 16s the patch passed +1 compile 0m 15s the patch passed -1 javac 0m 15s hadoop-hdfs-project_hadoop-hdfs-httpfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 checkstyle 0m 11s the patch passed +1 mvnsite 2m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 30s the patch passed +1 javadoc 0m 11s the patch passed       Other Tests +1 unit 3m 24s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 27m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12139 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877526/HDFS-12139.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e5bf79f886f7 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 02b141a Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HDFS-Build/20302/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20302/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20302/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

          People

          • Assignee:
            yzhangal Yongjun Zhang
            Reporter:
            yzhangal Yongjun Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development