Details
-
Bug
-
Status: Closed
-
Minor
-
Resolution: Fixed
-
None
-
None
-
None
Description
The credentials attached to a job are lost when CrunchOutputs builds a new Job in getJob.
For example, the following scenario fails:
MyTarget.java
class MyTarget implements MapReduceTarget { void configureForMapReduce(Job job, PType<?> ptype, Path outputPath, String name) { job.getCredentials().addSecretKey("myPassword", "secret".getBytes()); CrunchOutputs.addNamedOutput(job, name, FormatBundle.forOutput(MyOutputFormat.class), Object.class); } //.. }
MyOutputFormat.java
class MyOutputFormat extends OutputFormat<Object,Object> { public RecordWriter<Object, Object> getRecordWriter(TaskAttemptContext context) throws IOException, InterruptedException { Assert.equals("secret".getBytes(), context.getCredentials().getSecretKey("myPassword")); } }
This happens because CrunchOutputs.getJob copies the baseConf using the base Configuration class and the base Configuration class doesn't have a notion of Credentials, so the credentials get dropped.
This issue only manifests itself when using the local runner. In an actual hadoop cluster the credentials will be inherited from UserGroupInformation.