Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15022

Add new RPC to transfer data block with external shell script across Datanode

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • datanode
    • None

    Description

      Replicating data block is expensive when some Datanodes are down, especially for slow storage. Add a new RPC to replicate block with external shell script across datanode. User can choose more effective way to copy block files.

      In our setup, Archive volume are configured to remote reliable storage. we just add a new link file in new datanode to the remote file when do replication.

       

      Attachments

        1. HDFS-15022.patch
          34 kB
          Yang Yun
        2. HDFS-15022.patch
          35 kB
          Yang Yun
        3. link_block_across_datanode.pdf
          160 kB
          Yang Yun

        Issue Links

          is a child of

          New Feature - A new feature of the product, which has yet to be developed. HDFS-14627 Improvements to make slow archive storage works on HDFS

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              hadoop_yangyun Yang Yun
              hadoop_yangyun Yang Yun
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: