Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-600

Credentials are lost in local runner mode

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      The credentials attached to a job are lost when CrunchOutputs builds a new Job in getJob.
      For example, the following scenario fails:

      MyTarget.java
      class MyTarget implements MapReduceTarget {
        void configureForMapReduce(Job job, PType<?> ptype, Path outputPath, String name) {
          job.getCredentials().addSecretKey("myPassword", "secret".getBytes());
          CrunchOutputs.addNamedOutput(job, name, FormatBundle.forOutput(MyOutputFormat.class), Object.class);
        }
        //..
      }
      
      MyOutputFormat.java
      class MyOutputFormat extends OutputFormat<Object,Object> {
        public RecordWriter<Object, Object> getRecordWriter(TaskAttemptContext context) throws IOException, InterruptedException {
          Assert.equals("secret".getBytes(), context.getCredentials().getSecretKey("myPassword"));
        }
      }
      

      This happens because CrunchOutputs.getJob copies the baseConf using the base Configuration class and the base Configuration class doesn't have a notion of Credentials, so the credentials get dropped.

      This issue only manifests itself when using the local runner. In an actual hadoop cluster the credentials will be inherited from UserGroupInformation.

        Attachments

        1. CRUNCH-600-c.patch
          5 kB
          Igor Bernstein

          Activity

            People

            • Assignee:
              mkwhitacre Micah Whitacre
              Reporter:
              igorbernstein Igor Bernstein
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: