[HIVE-16686] repl invocations of distcp needs additional handling

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: repl
• Labels: TODOC3.0

Release Note

This introduces parsing of additional parameters that are not directly used by hive, but are passed on to distcp when hive invokes it. We now introduce the ability to use the hive command to do "set" commands to pass along cli arguments to distcp.

Any parameter set as "set distcp.options.blah=''" will result in an extra "-blah" argument going into distcp, as well as any parameter set as "set distcp.options.foo='bar'" will result in an extra "-foo bar" argument going to distcp.

Currently, we always pass along "-update" and "-skipcrccheck" to distcp - that is retained as defaults if no distcp.options.* params are found. If they are found, then these options are not added by default, letting the user instead provide an excplicit list.

Note that all of these properties affect how distcp runs when it is launched by hive, but are not directly hive settings. Instead, hive will allow setting them through the use of the "set" command.

• Assignee: Sushanth Sowmyan
• Reporter: Sushanth Sowmyan
• Created: 16/May/17 22:05
• Updated: 23/May/18 00:00
• Resolved: 24/May/17 16:38

Description

When REPL LOAD invokes distcp, there needs to be a way for the user invoking REPL LOAD to pass on arguments to distcp. In addition, there is sometimes a need for distcp to be invoked from within an impersonated context, such as running as user "hdfs", asking distcp to preserve ownerships of individual files.

Issue Links

• is related to: HIVE-17289 EXPORT and IMPORT shouldn't perform distcp with doAs privileged user.
• is related to: HIVE-16918 Skip ReplCopyTask distcp for _metadata copying. Also enable -pb for distcp
• relates to: HIVE-17571 update sql standard authorization config whitelist to include distcp options for replication
• links to: Reviewboard link