Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4943

WebHdfsFileSystem does not work when original file path has encoded chars

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 1.2.0, 1.1.2, 2.0.4-alpha, 0.23.10
    • 2.1.0-beta, 0.23.10
    • webhdfs
    • None
    • Reviewed

    Description

      In HBase, the WAL (hlog) file name on hdfs is URL encoded. For example,

      hdtest010%2C60020%2C1371000602151.1371058984668

      When we use webhdfs client to access the hlog file via httpfs, it does not work in this case.

      $ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs
      Found 1 items
      rw-rr- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668

      $ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668
      Found 1 items
      rw-rr- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668

      $ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs
      Found 1 items
      rw-rr- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668
      $
      $ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668
      13/06/27 18:36:08 DEBUG web.WebHdfsFileSystem: Original exception is
      org.apache.hadoop.ipc.RemoteException:java.io.FileNotFoundException:File does not exist: /user/biadmin/hbase_hlogs/hdtest010,60020,1371000602151.1371058984668
      at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:299)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:104)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:641)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:538)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:468)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:662)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:673)
      at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:1365)
      at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1048)
      at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:987)
      at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:965)
      at org.apache.hadoop.fs.FsShell.ls(FsShell.java:573)
      at org.apache.hadoop.fs.FsShell.doall(FsShell.java:1571)
      at org.apache.hadoop.fs.FsShell.run(FsShell.java:1789)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
      ls: Cannot access webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668: No such file or directory.

      Attachments

        1. HDFS-4943-trunk.patch
          0.9 kB
          Jerry He
        2. HDFS-4943-trunk-v2.patch
          3 kB
          Jerry He
        3. HDFS-4943-b023.patch
          2 kB
          Robert Parker

        Activity

          People

            jinghe Jerry He
            jinghe Jerry He
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: