[HADOOP-18056] DistCp: Filter duplicates in the source paths

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.2
• Fix Version/s: 3.4.0, 3.3.2
• Component/s: tools/distcp
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.2
• Hadoop Flags: Reviewed
• Assignee: Ayush Saxena
• Reporter: Ayush Saxena
• Created: 24/Dec/21 08:58
• Updated: 11/Feb/24 10:10
• Resolved: 05/Jan/22 18:29

Description

Add a basic filtering to remove the exact duplicate paths exposed for copying.

In case two same srcPath say /tmp/file1 is passed in the list twice. DistCp fails with DuplicateFileException, post building the listing.

Would be better if we do a basic filtering of duplicate paths.

Issue Links

• is related to: HIVE-25787 Prevent duplicate paths in the fileList while adding an entry to NotifcationLog
• links to: GitHub Pull Request #3825