[HDFS-4649] Webhdfs cannot list large directories

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1
• Fix Version/s: 0.23.7, 2.0.4-alpha
• Component/s: namenode, security, webhdfs
• Target Version/s: 0.23.7, 2.1.0-beta
• Hadoop Flags: Reviewed
• Assignee: Daryn Sharp
• Reporter: Daryn Sharp
• Created: 28/Mar/13 22:56
• Updated: 12/May/16 18:16
• Resolved: 02/Apr/13 19:17

Description

Webhdfs returns malformed json for directories that exceed the conf dfs.ls.limit value. The streaming object returned by NamenodeWebhdfsMethods#getListingStream will repeatedly call getListing for each segment of the directory listing. getListingStream runs within the remote user's ugi and acquires the first segment of the directory, then returns a streaming object. The streaming object is later executed outside of the user's ugi. Luckily it runs as the host service principal (ie. host/namenode@REALM) so the result is permission denied for the "host" user:

org.apache.hadoop.security.AccessControlException: Permission denied: user=host, access=EXECUTE, inode="/path":someuser:group:drwx------

The exception causes the streamer to prematurely abort the json output leaving it malformed. Meanwhile, the client sees the cryptic:

java.lang.IllegalStateException: unexpected end of array
        at org.mortbay.util.ajax.JSON.parseArray(JSON.java:902)
        [...]
        at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:242)
        at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:441)
        at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:717)
        [...]