[CRUNCH-600] Credentials are lost in local runner mode

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Assignee: Micah Whitacre
• Reporter: Igor Bernstein
• Created: 10/Apr/16 19:40
• Updated: 08/May/16 04:14
• Resolved: 11/Apr/16 15:47

Description

The credentials attached to a job are lost when CrunchOutputs builds a new Job in getJob.
For example, the following scenario fails:

MyTarget.java

class MyTarget implements MapReduceTarget {
  void configureForMapReduce(Job job, PType<?> ptype, Path outputPath, String name) {
    job.getCredentials().addSecretKey("myPassword", "secret".getBytes());
    CrunchOutputs.addNamedOutput(job, name, FormatBundle.forOutput(MyOutputFormat.class), Object.class);
  }
  //..
}

MyOutputFormat.java

class MyOutputFormat extends OutputFormat<Object,Object> {
  public RecordWriter<Object, Object> getRecordWriter(TaskAttemptContext context) throws IOException, InterruptedException {
    Assert.equals("secret".getBytes(), context.getCredentials().getSecretKey("myPassword"));
  }
}

This happens because CrunchOutputs.getJob copies the baseConf using the base Configuration class and the base Configuration class doesn't have a notion of Credentials, so the credentials get dropped.

This issue only manifests itself when using the local runner. In an actual hadoop cluster the credentials will be inherited from UserGroupInformation.