[HIVE-16901] Distcp optimization - One distcp per ReplCopyTask

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 3.0.0
• Component/s: Hive, repl
• Labels: DR replication
• Hadoop Flags: Reviewed
• Assignee: Sankar Hariappan
• Reporter: Sankar Hariappan
• Created: 14/Jun/17 18:48
• Updated: 22/May/18 23:59
• Resolved: 05/Jul/17 05:52

Description

Currently, if a ReplCopyTask is created to copy a list of files, then distcp is invoked for each and every file. Instead, need to pass the list of source files to be copied to distcp tool which basically copies the files in parallel and hence gets lot of performance gain.

Issue Links

• is a clone of: HIVE-16900 optimization to give distcp a list of input files to copy to a destination target directory during repl load
• links to: GitHub Pull Request #200