Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13916

Distcp SnapshotDiff to support WebHDFS

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.1, 3.1.1
    • 3.4.0
    • distcp, webhdfs
    • Reviewed
    • Patch

    Description

      ljain has worked on the JIRA: HDFS-13052 to provide the possibility to make DistCP of SnapshotDiff with WebHDFSFileSystem. However, in the patch, there is no modification for the real java class which is used by launching the command "hadoop distcp ..."

       

      You can check in the latest version here:

      https://github.com/apache/hadoop/blob/branch-3.1.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java#L96-L100

      In the method "preSyncCheck" of the class "DistCpSync", we still check if the file system is DFS. 

      So I propose to change the class DistCpSync in order to take into consideration what was committed by Lokesh Jain.

      Attachments

        1. HDFS-13916.patch
          11 kB
          Xun REN
        2. HDFS-13916.007.patch
          19 kB
          Masatake Iwasaki
        3. HDFS-13916.006.patch
          19 kB
          Wei-Chiu Chuang
        4. HDFS-13916.005.patch
          23 kB
          Xun REN
        5. HDFS-13916.004.patch
          23 kB
          Xun REN
        6. HDFS-13916.003.patch
          23 kB
          Xun REN
        7. HDFS-13916.002.patch
          11 kB
          Xun REN

        Issue Links

          Activity

            People

              renxunsaky Xun REN
              renxunsaky Xun REN
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: