Uploaded image for project: 'Crunch (Retired)'
  1. Crunch (Retired)
  2. CRUNCH-675

HFileTarget should use DistCp when source and destination are in different filesystems

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.0.0
    • None
    • None

    Description

      For jobs using the HFileTarget we have found that running a DistCp to handleOutputs at the conclusion of a job, when the source and destination are in different filesystems, provides a substantial performance benefit compared to a single-threaded data copy. This enhancement is similar to what was suggested in CRUNCH-660.

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. CRUNCH-660 FileTargetImpl uses Distcp vs FileUtils.copy

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              noslowerdna Andrew Olson
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: