Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4943

WebHdfsFileSystem does not work when original file path has encoded chars

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 1.1.2, 2.0.4-alpha, 0.23.10
    • Fix Version/s: 2.1.0-beta, 0.23.10
    • Component/s: webhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In HBase, the WAL (hlog) file name on hdfs is URL encoded. For example,

      hdtest010%2C60020%2C1371000602151.1371058984668

      When we use webhdfs client to access the hlog file via httpfs, it does not work in this case.

      $ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs
      Found 1 items
      rw-rr- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668

      $ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668
      Found 1 items
      rw-rr- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668

      $ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs
      Found 1 items
      rw-rr- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668
      $
      $ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668
      13/06/27 18:36:08 DEBUG web.WebHdfsFileSystem: Original exception is
      org.apache.hadoop.ipc.RemoteException:java.io.FileNotFoundException:File does not exist: /user/biadmin/hbase_hlogs/hdtest010,60020,1371000602151.1371058984668
      at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:299)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:104)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:641)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:538)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:468)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:662)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:673)
      at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:1365)
      at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1048)
      at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:987)
      at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:965)
      at org.apache.hadoop.fs.FsShell.ls(FsShell.java:573)
      at org.apache.hadoop.fs.FsShell.doall(FsShell.java:1571)
      at org.apache.hadoop.fs.FsShell.run(FsShell.java:1789)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
      ls: Cannot access webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668: No such file or directory.

        Attachments

        1. HDFS-4943-trunk.patch
          0.9 kB
          Jerry He
        2. HDFS-4943-trunk-v2.patch
          3 kB
          Jerry He
        3. HDFS-4943-b023.patch
          2 kB
          Robert Parker

          Activity

            People

            • Assignee:
              jinghe Jerry He
              Reporter:
              jinghe Jerry He
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: