Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4649

Webhdfs cannot list large directories

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1
    • Fix Version/s: 0.23.7, 2.0.4-alpha
    • Component/s: namenode, security, webhdfs
    • Labels:
      None

      Description

      Webhdfs returns malformed json for directories that exceed the conf dfs.ls.limit value. The streaming object returned by NamenodeWebhdfsMethods#getListingStream will repeatedly call getListing for each segment of the directory listing. getListingStream runs within the remote user's ugi and acquires the first segment of the directory, then returns a streaming object. The streaming object is later executed outside of the user's ugi. Luckily it runs as the host service principal (ie. host/namenode@REALM) so the result is permission denied for the "host" user:

      org.apache.hadoop.security.AccessControlException: Permission denied: user=host, access=EXECUTE, inode="/path":someuser:group:drwx------
      

      The exception causes the streamer to prematurely abort the json output leaving it malformed. Meanwhile, the client sees the cryptic:

      java.lang.IllegalStateException: unexpected end of array
              at org.mortbay.util.ajax.JSON.parseArray(JSON.java:902)
              [...]
              at
      org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:242)
              at
      org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:441)
              at
      org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:717)
              [...]
      

        Attachments

        1. HDFS-4649.patch
          6 kB
          Daryn Sharp
        2. HDFS-4649.branch-23.patch
          3 kB
          Daryn Sharp
        3. HDFS-4649.branch-23.patch
          6 kB
          Daryn Sharp

          Activity

            People

            • Assignee:
              daryn Daryn Sharp
              Reporter:
              daryn Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: