Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12090 Handling writes from HDFS to Provided storages
  3. HDFS-13940

Implement Multipart-aware distcp

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.3.0
    • None
    • fs
    • None

    Description

      the per block upload would permit a high-performance distcp

      • within the same FS
      • when writing to object stores with multipart uploads
        Implement this, if initially as a proof of concept to validate the completeness of the API

      One thing to consider is throttling upload bandwidth better: there's the potential here to overload the long-haul links more here, even when uploading a few files (as the peark bandwidth which a single file being copied ma use isO(blocks))

      Attachments

        Issue Links

          supercedes

          New Feature - A new feature of the product, which has yet to be developed. HDFS-8878 An HDFS built-in DistCp

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link Gobblin-Distcp

          Activity

            People

              Unassigned Unassigned
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: