[HADOOP-18117] Add an option to preserve root directory permissions

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: tools
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Mohanad Elsafty
• Reporter: Mohanad Elsafty
• Created: 09/Feb/22 08:22
• Updated: 11/Feb/24 10:23
• Resolved: 18/Feb/22 11:13

Description

As mentioned in https://issues.apache.org/jira/browse/HADOOP-15211

 

If -update or -overwrite is being passed when distcp used, the root directory will be skipped in two occasions (CopyListing#doBuildListing & CopyCommitter#preserveFileAttributesForDirectories), which will ignore root directory's attributes.

 

We face the same issue when distcp huge data between clusters and it takes too much effort to update root directories attributes manually.

 

From the earlier ticket it's obvious why this behaviour is there, but sometime we need to enforce root directory update hence I will add a new option for distcp to enable someone (who understands the need of this and know what they are doing) to enforce the update of root directory's attributes (permissions, ownership, ...)

 

It should be simple one, something like this

$ hadoop distcp -p -update -updateRootDirectoryAttributes /a/b/c /a/b/d 

This behaviour is optional and will be false by default. (it should not affect existing distcp users).

Issue Links

• fixes: HADOOP-15211 Distcp update not preserving root directory permissions
• links to: GitHub Pull Request #3970