Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7535

Utilize Snapshot diff report for distcp

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: distcp, snapshots
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently HDFS snapshot diff report can identify file/directory creation, deletion, rename and modification under a snapshottable directory. We can use the diff report for distcp between the primary cluster and a backup cluster to avoid unnecessary data copy. This is especially useful when there is a big directory rename happening in the primary cluster: the current distcp cannot detect the rename op thus this rename usually leads to large amounts of real data copy.

      More details of the approach will come in the first comment.

        Attachments

        1. HDFS-7535.004.patch
          39 kB
          Jing Zhao
        2. HDFS-7535.003.patch
          33 kB
          Jing Zhao
        3. HDFS-7535.002.patch
          33 kB
          Jing Zhao
        4. HDFS-7535.001.patch
          33 kB
          Jing Zhao
        5. HDFS-7535.000.patch
          21 kB
          Jing Zhao

          Activity

            People

            • Assignee:
              jingzhao Jing Zhao
              Reporter:
              jingzhao Jing Zhao
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: