Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8828

Utilize Snapshot diff report to build diff copy list in distcp



    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: distcp, snapshots
    • Labels:
    • Hadoop Flags:


      Some users reported huge time cost to build file copy list in distcp. (30 hours for 1.6M files). We can leverage snapshot diff report to build file copy list including files/dirs which are changes only between two snapshots (or a snapshot and a normal dir). It speed up the process in two folds: 1. less copy list building time. 2. less file copy MR jobs.

      HDFS snapshot diff report provide information about file/directory creation, deletion, rename and modification between two snapshots or a snapshot and a normal directory. HDFS-7535 synchronize deletion and rename, then fallback to the default distcp. So it still relies on default distcp to building complete list of files under the source dir. This patch only puts creation and modification files into the copy list based on snapshot diff report. We can minimize the number of files to copy.


        1. HDFS-8828.001.patch
          27 kB
          Yufei Gu
        2. HDFS-8828.002.patch
          27 kB
          Yufei Gu
        3. HDFS-8828.003.patch
          27 kB
          Yufei Gu
        4. HDFS-8828.004.patch
          30 kB
          Yufei Gu
        5. HDFS-8828.005.patch
          32 kB
          Yufei Gu
        6. HDFS-8828.006.patch
          41 kB
          Yufei Gu
        7. HDFS-8828.007.patch
          41 kB
          Yufei Gu
        8. HDFS-8828.008.patch
          53 kB
          Yufei Gu
        9. HDFS-8828.009.patch
          54 kB
          Yufei Gu
        10. HDFS-8828.010.patch
          55 kB
          Yufei Gu
        11. HDFS-8828.011.patch
          54 kB
          Yufei Gu

          Issue Links



              • Assignee:
                yufeigu Yufei Gu
                yufeigu Yufei Gu
              • Votes:
                0 Vote for this issue
                12 Start watching this issue


                • Created: