Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4649

Webhdfs cannot list large directories

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.23.0, 2.0.0-alpha, 3.0.0-alpha1
    • 0.23.7, 2.0.4-alpha
    • namenode, security, webhdfs
    • None

    Description

      Webhdfs returns malformed json for directories that exceed the conf dfs.ls.limit value. The streaming object returned by NamenodeWebhdfsMethods#getListingStream will repeatedly call getListing for each segment of the directory listing. getListingStream runs within the remote user's ugi and acquires the first segment of the directory, then returns a streaming object. The streaming object is later executed outside of the user's ugi. Luckily it runs as the host service principal (ie. host/namenode@REALM) so the result is permission denied for the "host" user:

      org.apache.hadoop.security.AccessControlException: Permission denied: user=host, access=EXECUTE, inode="/path":someuser:group:drwx------
      

      The exception causes the streamer to prematurely abort the json output leaving it malformed. Meanwhile, the client sees the cryptic:

      java.lang.IllegalStateException: unexpected end of array
              at org.mortbay.util.ajax.JSON.parseArray(JSON.java:902)
              [...]
              at
      org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:242)
              at
      org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:441)
              at
      org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:717)
              [...]
      

      Attachments

        1. HDFS-4649.patch
          6 kB
          Daryn Sharp
        2. HDFS-4649.branch-23.patch
          3 kB
          Daryn Sharp
        3. HDFS-4649.branch-23.patch
          6 kB
          Daryn Sharp

        Activity

          People

            daryn Daryn Sharp
            daryn Daryn Sharp
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: