[HIVE-14848] S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.0
• Component/s: Hive
• Target Version/s: 3.0.0
• Assignee: Jaume M
• Reporter: Sergio Peña
• Created: 27/Sep/16 20:19
• Updated: 16/Aug/18 14:30

Description

When S3 credentials are included in hive-site.xml, then MR jobs that need to read
data from S3 cannot use them because S3 values are stripped from the Job configuration
object before submitting the MR job.

@Override
public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext driverContext) {
  ...

  conf.stripHiddenConfigurations(job);
  this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
}

A nice to have (available on hadoop 2.9.0) is an MR mapreduce.job.redacted-properties that can be used to hide this list on the MR side (such as history server UI) to allow MR run the job without issues.

UPDATE:
Change the call to stripHiddenConfigurations() in ql/exec/tez/DagUtils.createConfiguration(), because this is currently broken for running hive-blobstore suite against Tez

Issue Links

• is related to: HIVE-14588 Add S3 credentials to the hidden configuration variable supported on HIVE-14207