[HDFS-4943] WebHdfsFileSystem does not work when original file path has encoded chars

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.1.2, 2.0.4-alpha, 0.23.10
• Fix Version/s: 2.1.0-beta, 0.23.10
• Component/s: webhdfs
• Hadoop Flags: Reviewed
• Assignee: Jerry He
• Reporter: Jerry He
• Created: 28/Jun/13 21:50
• Updated: 27/Aug/13 22:07
• Resolved: 02/Jul/13 15:01

Description

In HBase, the WAL (hlog) file name on hdfs is URL encoded. For example,

hdtest010%2C60020%2C1371000602151.1371058984668

When we use webhdfs client to access the hlog file via httpfs, it does not work in this case.

$ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs
Found 1 items
rw-rr- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668

$ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668
Found 1 items
rw-rr- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668

$ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs
Found 1 items
rw-rr- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668
$
$ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668
13/06/27 18:36:08 DEBUG web.WebHdfsFileSystem: Original exception is
org.apache.hadoop.ipc.RemoteException:java.io.FileNotFoundException:File does not exist: /user/biadmin/hbase_hlogs/hdtest010,60020,1371000602151.1371058984668
at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:299)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:104)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:641)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:538)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:468)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:662)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:673)
at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:1365)
at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1048)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:987)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:965)
at org.apache.hadoop.fs.FsShell.ls(FsShell.java:573)
at org.apache.hadoop.fs.FsShell.doall(FsShell.java:1571)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:1789)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
ls: Cannot access webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668: No such file or directory.