[HIVE-20521] HS2 doAs=true has permission issue with hadoop.tmp.dir, with MR and S3A filesystem

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.0.0
• Fix Version/s: 4.0.0-alpha-1
• Assignee: Thejas Nair
• Reporter: Thejas Nair
• Created: 08/Sep/18 00:06
• Updated: 17/Nov/22 08:54
• Resolved: 11/Sep/18 15:45

Description

This is a result of changes in HIVE-18858.
As described by puneetj in HIVE-18858 -

This seems to have broken working scenarios with Hive MR. We now see hadoop.tmp.dir is always set to /tmp/hadoop-hive (in job.xml). This creates problems on a multi-tenant hadoop cluster since ownership of tmp folder is set to the user who executes the jobs first and other users fails to write to tmp folder.

E.g. User1 run job and /tmp/hadoop-hive is created on worker node with ownership to user1 and sibsequently user2 tries to run a job and job fails due to no write permission on /tmp/hadoop-hive/

Old behavior allowed multiple tenants to write to their respective tmp folders which was secure and contention free. User1 - /tmp/hadoop-user1, User2 - /tmp/hadoop-user2.

 
The change in HIVE-18858 causes variable expansion to happen in HiveServer2, while it was happening in the tasks (ExecMapper, ExecReducer) before that change. THis causes

"/tmp/hadoop-{user.name}"

to be expanded as /tmp/hadoop-hive instead of /tmp/hadoop-user1

Issue Links

• is broken by: HIVE-18858 System properties in job configuration not resolved when submitting MR job
• is related to: HADOOP-15722 regression: Hadoop 2.7.7 release breaks spark submit