Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4649

Webhdfs cannot list large directories

    Details

      Description

      Webhdfs returns malformed json for directories that exceed the conf dfs.ls.limit value. The streaming object returned by NamenodeWebhdfsMethods#getListingStream will repeatedly call getListing for each segment of the directory listing. getListingStream runs within the remote user's ugi and acquires the first segment of the directory, then returns a streaming object. The streaming object is later executed outside of the user's ugi. Luckily it runs as the host service principal (ie. host/namenode@REALM) so the result is permission denied for the "host" user:

      org.apache.hadoop.security.AccessControlException: Permission denied: user=host, access=EXECUTE, inode="/path":someuser:group:drwx------
      

      The exception causes the streamer to prematurely abort the json output leaving it malformed. Meanwhile, the client sees the cryptic:

      java.lang.IllegalStateException: unexpected end of array
              at org.mortbay.util.ajax.JSON.parseArray(JSON.java:902)
              [...]
              at
      org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:242)
              at
      org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:441)
              at
      org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:717)
              [...]
      
      1. HDFS-4649.branch-23.patch
        3 kB
        Daryn Sharp
      2. HDFS-4649.branch-23.patch
        6 kB
        Daryn Sharp
      3. HDFS-4649.patch
        6 kB
        Daryn Sharp

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1390 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1390/)
        HDFS-4649. Webhdfs cannot list large directories. Contributed by Daryn Sharp. (Revision 1463698)

        Result = SUCCESS
        kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463698
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1390 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1390/ ) HDFS-4649 . Webhdfs cannot list large directories. Contributed by Daryn Sharp. (Revision 1463698) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463698 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1363 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1363/)
        HDFS-4649. Webhdfs cannot list large directories. Contributed by Daryn Sharp. (Revision 1463698)

        Result = FAILURE
        kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463698
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1363 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1363/ ) HDFS-4649 . Webhdfs cannot list large directories. Contributed by Daryn Sharp. (Revision 1463698) Result = FAILURE kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463698 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #572 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/572/)
        HDFS-4649. Webhdfs cannot list large directories. Contributed by Daryn Sharp. (Revision 1463700)

        Result = UNSTABLE
        kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463700
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #572 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/572/ ) HDFS-4649 . Webhdfs cannot list large directories. Contributed by Daryn Sharp. (Revision 1463700) Result = UNSTABLE kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463700 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #174 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/174/)
        HDFS-4649. Webhdfs cannot list large directories. Contributed by Daryn Sharp. (Revision 1463698)

        Result = SUCCESS
        kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463698
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #174 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/174/ ) HDFS-4649 . Webhdfs cannot list large directories. Contributed by Daryn Sharp. (Revision 1463698) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463698 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3550 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3550/)
        HDFS-4649. Webhdfs cannot list large directories. Contributed by Daryn Sharp. (Revision 1463698)

        Result = SUCCESS
        kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463698
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3550 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3550/ ) HDFS-4649 . Webhdfs cannot list large directories. Contributed by Daryn Sharp. (Revision 1463698) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463698 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
        Hide
        Kihwal Lee added a comment -

        I've committed this to branch-0.23, branch-2 and trunk.

        Show
        Kihwal Lee added a comment - I've committed this to branch-0.23, branch-2 and trunk.
        Hide
        Kihwal Lee added a comment -

        The test failure was not caused by this patch and will be fixed in HDFS-4653.
        +1 The patch looks good to me.

        Show
        Kihwal Lee added a comment - The test failure was not caused by this patch and will be fixed in HDFS-4653 . +1 The patch looks good to me.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12576119/HDFS-4649.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.fs.TestFcHdfsSymlink

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4168//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4168//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576119/HDFS-4649.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.TestFcHdfsSymlink +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4168//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4168//console This message is automatically generated.
        Hide
        Daryn Sharp added a comment -

        Added some comments, added tests. The trunk/23 patches are identical except for the imports section in the test file.

        Show
        Daryn Sharp added a comment - Added some comments, added tests. The trunk/23 patches are identical except for the imports section in the test file.
        Hide
        Daryn Sharp added a comment -

        Save the user's ugi, and reuse it in the streamer object. Simplified the code to loop over the segments. I need to write tests, but it has been successfully tested on a secure cluster blocked by this issue.

        The patch is for 23, but I expect the trunk/2 patches to be similar if not identical. I will finish the patches tomorrow.

        Show
        Daryn Sharp added a comment - Save the user's ugi, and reuse it in the streamer object. Simplified the code to loop over the segments. I need to write tests, but it has been successfully tested on a secure cluster blocked by this issue. The patch is for 23, but I expect the trunk/2 patches to be similar if not identical. I will finish the patches tomorrow.

          People

          • Assignee:
            Daryn Sharp
            Reporter:
            Daryn Sharp
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development