Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1967

hadoop dfs -ls, -get, -mv command's source/destination URI are inconsistent

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.14.1
    • 0.17.0
    • None
    • None

    Description

      While specifying source/destination path for hadoop dfs -ls, -get, -mv, -cp commands, we have some inconsistency related to 'hdfs://' scheme.

      Particularly, few of the commands accept both formats
      [1] hdfs:///user/lohit/testfile
      [2] hdfs://myhost:8020/user/lohit/testfile

      and few commands accept only paths, which have authority (host:port)
      [2] hdfs://myhost:8020/user/lohit/testfile

      below are examples
      hadoop dfs -ls (works for both formats)

      [lohit@krygw1000 ~]$ hadoop dfs -ls hdfs://kry-nn1:8020/user/lohit/ranges
      Found 1 items
      /user/lohit/ranges <r 3> 24 1970-01-01 00:00
      [lohit@krygw1000 ~]$ hadoop dfs -ls hdfs:///user/lohit/ranges
      Found 1 items

      hadoop dfs -get (works for only format [2])

      [lohit@krygw1000 ~]$ hadoop dfs -get hdfs:///user/lohit/ranges .
      Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
      hdfs:/user/lohit/ranges, expected: hdfs://kry-nn1:8020
      at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:204)
      at
      org.apache.hadoop.dfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:108)
      at
      org.apache.hadoop.dfs.DistributedFileSystem.getPath(DistributedFileSystem.java:104)
      at
      org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:319)
      at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:423)
      at org.apache.hadoop.fs.FsShell.copyToLocal(FsShell.java:177)
      at org.apache.hadoop.fs.FsShell.copyToLocal(FsShell.java:155)
      at org.apache.hadoop.fs.FsShell.run(FsShell.java:1233)
      at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
      at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
      [lohit@krygw1000 ~]$ hadoop dfs -get hdfs://kry-nn1:8020/user/lohit/ranges .
      [lohit@krygw1000 ~]$ ls ./ranges
      ./ranges
      [lohit@krygw1000 ~]$

      hadoop dfs -mv / -cp command. source path accepts both format [1] and [2], while destination accepts only [2].

      [lohit@krygw1000 ~]$ hadoop dfs -cp hdfs://kry-nn1:8020/user/lohit/ranges.test2
      hdfs:///user/lohit/ranges.test
      Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
      hdfs:/user/lohit/ranges.test, expected: hdfs://kry-nn1:8020
      at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:204)
      at
      org.apache.hadoop.dfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:108)
      at
      org.apache.hadoop.dfs.DistributedFileSystem.getPath(DistributedFileSystem.java:104)
      at
      org.apache.hadoop.dfs.DistributedFileSystem.exists(DistributedFileSystem.java:162)
      at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:269)
      at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
      at org.apache.hadoop.fs.FsShell.copy(FsShell.java:691)
      at org.apache.hadoop.fs.FsShell.copy(FsShell.java:727)
      at org.apache.hadoop.fs.FsShell.run(FsShell.java:1260)
      at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
      at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
      [lohit@krygw1000 ~]$ hadoop dfs -cp hdfs:///user/lohit/ranges.test2
      hdfs://kry-nn1:8020/user/lohit/ranges.test
      [lohit@krygw1000 ~]$

      We should have a consistent URI naming convention across all commands.

      Attachments

        1. HADOOP-1967_1.patch
          3 kB
          Mahadev Konar
        2. HADOOP-1967_2.patch
          7 kB
          Doug Cutting
        3. HADOOP-1967_3.patch
          7 kB
          Doug Cutting
        4. HADOOP-1967_4.patch
          6 kB
          Doug Cutting

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cutting Doug Cutting
            lohit Lohit Vijaya Renu
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment