Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2139

Fast copy for HDFS.

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      There is a need to perform fast file copy on HDFS. The fast copy mechanism for a file works as
      follows :

      1) Query metadata for all blocks of the source file.

      2) For each block 'b' of the file, find out its datanode locations.

      3) For each block of the file, add an empty block to the namesystem for
      the destination file.

      4) For each location of the block, instruct the datanode to make a local
      copy of that block.

      5) Once each datanode has copied over its respective blocks, they
      report to the namenode about it.

      6) Wait for all blocks to be copied and exit.

      This would speed up the copying process considerably by removing top of
      the rack data transfers.

      Note : An extra improvement, would be to instruct the datanode to create a
      hardlink of the block file if we are copying a block on the same datanode

        Attachments

        1. HDFS-2139.patch
          50 kB
          yunjiong zhao
        2. HDFS-2139.patch
          55 kB
          yunjiong zhao
        3. HDFS-2139-For-2.7.1.patch
          96 kB
          Liu Junhong

        Issue Links

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

              Dates

              • Created:
                Updated:

              Time Tracking

              Estimated:
              Original Estimate - 168h
              168h
              Remaining:
              Remaining Estimate - 168h
              168h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Issue deployment