Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5899

Support incremental data copy in DistCp

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.5.0
    • Component/s: distcp
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently when doing distcp with -update option, for two files with the same file names but with different file length or checksum, we overwrite the whole file. It will be good if we can detect the case where (sourceFile = targetFile + appended_data), and only transfer the appended data segment to the target. This will be very useful if we're doing incremental distcp.

        Attachments

        1. HADOOP-10608.000.patch
          56 kB
          Jing Zhao
        2. HADOOP-10608.001.patch
          56 kB
          Jing Zhao
        3. MAPREDUCE-5899.002.patch
          59 kB
          Jing Zhao
        4. MAPREDUCE-5899.002.patch
          59 kB
          Jing Zhao

          Issue Links

            Activity

              People

              • Assignee:
                jingzhao Jing Zhao
                Reporter:
                jingzhao Jing Zhao
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: