[HBASE-20579] Improve snapshot manifest copy in ExportSnapshot

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.4.0, 1.5.0, 2.0.0
• Fix Version/s: 3.0.0, 2.1.0
• Component/s: mapreduce
• Hadoop Flags: Reviewed

Release Note

This patch adds an FSUtil.copyFilesParallel() to help copy files in parallel, and it will return all the paths of directories and files traversed. Thus when we copy manifest in ExportSnapshot, we can copy reference files concurrently and use the paths it returns to help setOwner and setPermission.
The size of thread pool is determined by the configuration snapshot.export.copy.references.threads, and its default value is the number of runtime available processors.

• Assignee: Jingyun Tian
• Reporter: Jingyun Tian
• Created: 14/May/18 04:06
• Updated: 03/Jun/18 13:15
• Resolved: 03/Jun/18 02:32

Description

ExportSnapshot need to copy snapshot manifest to destination cluster first, then setOwner and setPermission for those paths. But it's done with one thread, which lead to a long time to submit the job if your snapshot is big. I tried to make them processing in parallel, which can reduce the total time of submitting dramatically.

Issue Links

• links to: Review Board (master)