Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4943

WebHdfsFileSystem does not work when original file path has encoded chars

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 1.1.2, 2.0.4-alpha, 0.23.10
    • Fix Version/s: 2.1.0-beta, 0.23.10
    • Component/s: webhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In HBase, the WAL (hlog) file name on hdfs is URL encoded. For example,

      hdtest010%2C60020%2C1371000602151.1371058984668

      When we use webhdfs client to access the hlog file via httpfs, it does not work in this case.

      $ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs
      Found 1 items
      rw-rr- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668

      $ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668
      Found 1 items
      rw-rr- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668

      $ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs
      Found 1 items
      rw-rr- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668
      $
      $ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668
      13/06/27 18:36:08 DEBUG web.WebHdfsFileSystem: Original exception is
      org.apache.hadoop.ipc.RemoteException:java.io.FileNotFoundException:File does not exist: /user/biadmin/hbase_hlogs/hdtest010,60020,1371000602151.1371058984668
      at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:299)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:104)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:641)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:538)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:468)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:662)
      at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:673)
      at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:1365)
      at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1048)
      at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:987)
      at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:965)
      at org.apache.hadoop.fs.FsShell.ls(FsShell.java:573)
      at org.apache.hadoop.fs.FsShell.doall(FsShell.java:1571)
      at org.apache.hadoop.fs.FsShell.run(FsShell.java:1789)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895)
      ls: Cannot access webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668: No such file or directory.

      1. HDFS-4943-trunk-v2.patch
        3 kB
        Jerry He
      2. HDFS-4943-trunk.patch
        0.9 kB
        Jerry He
      3. HDFS-4943-b023.patch
        2 kB
        Robert Parker

        Activity

        Jerry He created issue -
        Jerry He made changes -
        Field Original Value New Value
        Attachment HDFS-4943-trunk.patch [ 12590107 ]
        Jerry He made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Jerry He made changes -
        Attachment HDFS-4943-trunk-v2.patch [ 12590344 ]
        Tsz Wo Nicholas Sze made changes -
        Hadoop Flags Reviewed [ 10343 ]
        Assignee Jerry He [ jinghe ]
        Fix Version/s 2.1.0-beta [ 12324031 ]
        Tsz Wo Nicholas Sze made changes -
        Fix Version/s 2.1.0-beta [ 12324031 ]
        Tsz Wo Nicholas Sze made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Robert Parker made changes -
        Attachment HDFS-4943-b023.patch [ 12591224 ]
        Kihwal Lee made changes -
        Affects Version/s 0.23.10 [ 12324660 ]
        Kihwal Lee made changes -
        Fix Version/s 0.23.10 [ 12324660 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Jerry He
            Reporter:
            Jerry He
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development