Pig
  1. Pig
  2. PIG-3507

It fails to run pig in local mode on a Kerberos enabled Hadoop cluster

    Details

    • Type: Bug Bug
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.10.0, 0.11
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      It fails to run pig in local mode on a Kerberos enabled Hadoop cluster

      Command
      pig -x local <pig script>

      Pig script
      A = load '/etc/passwd';
      dump A;

      Root cause
      When running pig in local mode, jobConf in HExecutionEngine is initiated with core-default.xml (hadoop.security.authentication = simple), mapred-default.xml, and yarn-default.xml. However, the settings are not passed to UserGroupInformation. That's why obtainTokensForNamenodesInternal() is called from obtainTokensForNamenodes(), and causes the exception to happen.

      public static void obtainTokensForNamenodes(Credentials credentials, Path[] ps, Configuration conf) throws IOException {
          if (!UserGroupInformation.isSecurityEnabled()) {
              return;
          }
          obtainTokensForNamenodesInternal(credentials, ps, conf);
      }	
      

      Error
      Pig Stack Trace
      ---------------
      ERROR 6000: Output Location Validation Failed for: 'file:/tmp/temp-308998488/tmp-2025176494 More info to follow:
      Can't get JT Kerberos principal for use as renewer

      org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias A
      at org.apache.pig.PigServer.openIterator(PigServer.java:841)
      at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
      at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
      at org.apache.pig.Main.run(Main.java:604)
      at org.apache.pig.Main.main(Main.java:157)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
      Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
      at org.apache.pig.PigServer.storeEx(PigServer.java:940)
      at org.apache.pig.PigServer.store(PigServer.java:903)
      at org.apache.pig.PigServer.openIterator(PigServer.java:816)
      ... 12 more
      Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: Output Location Validation Failed for: 'file:/tmp/temp-308998488/tmp-2025176494 More info to follow:
      Can't get JT Kerberos principal for use as renewer
      at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95)
      at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
      at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
      at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
      at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
      at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
      at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
      at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:288)
      at org.apache.pig.PigServer.compilePp(PigServer.java:1327)
      at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1252)
      at org.apache.pig.PigServer.storeEx(PigServer.java:936)
      ... 14 more
      Caused by: java.io.IOException: Can't get JT Kerberos principal for use as renewer
      at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:129)
      at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:111)
      at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:85)
      at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:127)
      at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80)
      ... 24 more
      ================================================================================

        Activity

        Aniket Mokashi made changes -
        Fix Version/s 0.14.0 [ 12326954 ]
        Fix Version/s 0.13.0 [ 12324971 ]
        Rohini Palaniswamy made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Cheolsoo Park made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Cheolsoo Park made changes -
        Fix Version/s 0.13.0 [ 12324971 ]
        Fix Version/s 0.10.0 [ 12316246 ]
        Cheolsoo Park made changes -
        Assignee chiyang [ chiyang ]
        chiyang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        chiyang made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        chiyang made changes -
        Attachment PIG-3507.patch [ 12607306 ]
        chiyang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s 0.10.0 [ 12316246 ]
        chiyang made changes -
        Description It fails to run pig in local mode on a Kerberos enabled Hadoop cluster

        *Command*
        pig -x local <pig script>

        *Pig script*
        A = load '/etc/passwd';
        dump A;

        *Root cause*
        When running pig in local mode, jobConf in HExecutionEngine is initiated with core-default.xml (hadoop.security.authentication = simple), mapred-default.xml, and yarn-default.xml. However, the settings are not passed to UserGroupInformation. That's why obtainTokensForNamenodesInternal() is called in obtainTokensForNamenodes(), and causes the exception to happen.

        {noformat}
        public static void obtainTokensForNamenodes(Credentials credentials, Path[] ps, Configuration conf) throws IOException {
            if (!UserGroupInformation.isSecurityEnabled()) {
                return;
            }
            obtainTokensForNamenodesInternal(credentials, ps, conf);
        }
        {noformat}

        *Error*
        Pig Stack Trace
        ---------------
        ERROR 6000: Output Location Validation Failed for: 'file:/tmp/temp-308998488/tmp-2025176494 More info to follow:
        Can't get JT Kerberos principal for use as renewer

        org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias A
        at org.apache.pig.PigServer.openIterator(PigServer.java:841)
        at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
        at org.apache.pig.Main.run(Main.java:604)
        at org.apache.pig.Main.main(Main.java:157)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
        Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
        at org.apache.pig.PigServer.storeEx(PigServer.java:940)
        at org.apache.pig.PigServer.store(PigServer.java:903)
        at org.apache.pig.PigServer.openIterator(PigServer.java:816)
        ... 12 more
        Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: Output Location Validation Failed for: 'file:/tmp/temp-308998488/tmp-2025176494 More info to follow:
        Can't get JT Kerberos principal for use as renewer
        at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95)
        at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
        at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
        at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
        at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
        at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
        at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:288)
        at org.apache.pig.PigServer.compilePp(PigServer.java:1327)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1252)
        at org.apache.pig.PigServer.storeEx(PigServer.java:936)
        ... 14 more
        Caused by: java.io.IOException: Can't get JT Kerberos principal for use as renewer
        at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:129)
        at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:111)
        at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:85)
        at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:127)
        at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80)
        ... 24 more
        ================================================================================
        It fails to run pig in local mode on a Kerberos enabled Hadoop cluster

        *Command*
        pig -x local <pig script>

        *Pig script*
        A = load '/etc/passwd';
        dump A;

        *Root cause*
        When running pig in local mode, jobConf in HExecutionEngine is initiated with core-default.xml (hadoop.security.authentication = simple), mapred-default.xml, and yarn-default.xml. However, the settings are not passed to UserGroupInformation. That's why obtainTokensForNamenodesInternal() is called from obtainTokensForNamenodes(), and causes the exception to happen.

        {noformat}
        public static void obtainTokensForNamenodes(Credentials credentials, Path[] ps, Configuration conf) throws IOException {
            if (!UserGroupInformation.isSecurityEnabled()) {
                return;
            }
            obtainTokensForNamenodesInternal(credentials, ps, conf);
        }
        {noformat}

        *Error*
        Pig Stack Trace
        ---------------
        ERROR 6000: Output Location Validation Failed for: 'file:/tmp/temp-308998488/tmp-2025176494 More info to follow:
        Can't get JT Kerberos principal for use as renewer

        org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias A
        at org.apache.pig.PigServer.openIterator(PigServer.java:841)
        at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
        at org.apache.pig.Main.run(Main.java:604)
        at org.apache.pig.Main.main(Main.java:157)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
        Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
        at org.apache.pig.PigServer.storeEx(PigServer.java:940)
        at org.apache.pig.PigServer.store(PigServer.java:903)
        at org.apache.pig.PigServer.openIterator(PigServer.java:816)
        ... 12 more
        Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: Output Location Validation Failed for: 'file:/tmp/temp-308998488/tmp-2025176494 More info to follow:
        Can't get JT Kerberos principal for use as renewer
        at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95)
        at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
        at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
        at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
        at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
        at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
        at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:288)
        at org.apache.pig.PigServer.compilePp(PigServer.java:1327)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1252)
        at org.apache.pig.PigServer.storeEx(PigServer.java:936)
        ... 14 more
        Caused by: java.io.IOException: Can't get JT Kerberos principal for use as renewer
        at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:129)
        at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:111)
        at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:85)
        at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:127)
        at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80)
        ... 24 more
        ================================================================================
        chiyang made changes -
        Field Original Value New Value
        Affects Version/s 0.11 [ 12318878 ]
        chiyang created issue -

          People

          • Assignee:
            chiyang
            Reporter:
            chiyang
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development