Pig
  1. Pig
  2. PIG-3507

Pig fails to run in local mode on a Kerberos enabled Hadoop cluster

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0, 0.11
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It fails to run pig in local mode on a Kerberos enabled Hadoop cluster

      Command
      pig -x local <pig script>

      Pig script
      A = load '/etc/passwd';
      dump A;

      Root cause
      When running pig in local mode, jobConf in HExecutionEngine is initiated with core-default.xml (hadoop.security.authentication = simple), mapred-default.xml, and yarn-default.xml. However, the settings are not passed to UserGroupInformation. That's why obtainTokensForNamenodesInternal() is called from obtainTokensForNamenodes(), and causes the exception to happen.

      public static void obtainTokensForNamenodes(Credentials credentials, Path[] ps, Configuration conf) throws IOException {
          if (!UserGroupInformation.isSecurityEnabled()) {
              return;
          }
          obtainTokensForNamenodesInternal(credentials, ps, conf);
      }	
      

      Error
      Pig Stack Trace
      ---------------
      ERROR 6000: Output Location Validation Failed for: 'file:/tmp/temp-308998488/tmp-2025176494 More info to follow:
      Can't get JT Kerberos principal for use as renewer

      org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias A
      at org.apache.pig.PigServer.openIterator(PigServer.java:841)
      at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
      at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
      at org.apache.pig.Main.run(Main.java:604)
      at org.apache.pig.Main.main(Main.java:157)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
      Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias A
      at org.apache.pig.PigServer.storeEx(PigServer.java:940)
      at org.apache.pig.PigServer.store(PigServer.java:903)
      at org.apache.pig.PigServer.openIterator(PigServer.java:816)
      ... 12 more
      Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 6000: Output Location Validation Failed for: 'file:/tmp/temp-308998488/tmp-2025176494 More info to follow:
      Can't get JT Kerberos principal for use as renewer
      at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95)
      at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
      at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
      at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
      at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
      at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
      at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
      at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:288)
      at org.apache.pig.PigServer.compilePp(PigServer.java:1327)
      at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1252)
      at org.apache.pig.PigServer.storeEx(PigServer.java:936)
      ... 14 more
      Caused by: java.io.IOException: Can't get JT Kerberos principal for use as renewer
      at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:129)
      at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:111)
      at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:85)
      at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:127)
      at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80)
      ... 24 more
      ================================================================================

      1. PIG_3507_1.patch
        1 kB
        liyunzhang_intel
      2. PIG-3507.patch
        1 kB
        chiyang
      3. PIG-3507-2.patch
        0.9 kB
        liyunzhang_intel

        Activity

        Hide
        takeshi.miao added a comment -

        This patch works for our env with kerberos enabled cluster as well. I think it is valuable if someone suffer the same issue, can anyone help to review this patch ?

        Show
        takeshi.miao added a comment - This patch works for our env with kerberos enabled cluster as well. I think it is valuable if someone suffer the same issue, can anyone help to review this patch ?
        Hide
        Cheolsoo Park added a comment -

        Sorry for the late review.

        Since I don't have kerberos set up, I can't really reproduce the issue. But the patch itself looks reasonable to me. If I don't hear any objections, I will commit the patch on Monday. Please chime in if anyone has concerns.

        Show
        Cheolsoo Park added a comment - Sorry for the late review. Since I don't have kerberos set up, I can't really reproduce the issue. But the patch itself looks reasonable to me. If I don't hear any objections, I will commit the patch on Monday. Please chime in if anyone has concerns.
        Hide
        Cheolsoo Park added a comment -

        Committed to trunk. Thank you Chiyang!

        Show
        Cheolsoo Park added a comment - Committed to trunk. Thank you Chiyang!
        Hide
        Rohini Palaniswamy added a comment -

        This patch needs to be reverted if we are still supporting hadoop 0.23 as it will break compatibility.

        http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/src/core/org/apache/hadoop/security/UserGroupInformation.java?view=markup does not have setConfiguration method.

        Also I don't think the fix is correct. The problem that is happening is their core-site.xml contains hadoop.security.authentication = kerberos and that is getting picked up as it is loaded as a default resource in UserGroupInformation's configuration. We should fix the bin/pig script to not load HADOOP_CONF_DIR in local mode.

        Show
        Rohini Palaniswamy added a comment - This patch needs to be reverted if we are still supporting hadoop 0.23 as it will break compatibility. http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/src/core/org/apache/hadoop/security/UserGroupInformation.java?view=markup does not have setConfiguration method. Also I don't think the fix is correct. The problem that is happening is their core-site.xml contains hadoop.security.authentication = kerberos and that is getting picked up as it is loaded as a default resource in UserGroupInformation's configuration. We should fix the bin/pig script to not load HADOOP_CONF_DIR in local mode.
        Hide
        Cheolsoo Park added a comment -

        Rohini Palaniswamy, my bad. Feel free to revert it.

        Show
        Cheolsoo Park added a comment - Rohini Palaniswamy , my bad. Feel free to revert it.
        Hide
        Rohini Palaniswamy added a comment -

        Thanks Cheolsoo Park. Reverting for now and reopening this jira till we find the actual root cause and decide on the appropriate fix. I am suspecting that the culprit in their case is that core-site.xml is being picked up in UserGroupInformation and that causes problem.

        Show
        Rohini Palaniswamy added a comment - Thanks Cheolsoo Park . Reverting for now and reopening this jira till we find the actual root cause and decide on the appropriate fix. I am suspecting that the culprit in their case is that core-site.xml is being picked up in UserGroupInformation and that causes problem.
        Hide
        liyunzhang_intel added a comment -

        The problem that pig can not work in local mode with kerberos, only exists in hadoop2.

        As the root cause of the bug description said:
        When running pig in local mode, jobConf in HExecutionEngine is initiated with core-default.xml (hadoop.security.authentication = simple), mapred-default.xml, and yarn-default.xml. However, the settings are not passed to UserGroupInformation. That's why obtainTokensForNamenodesInternal() is called from obtainTokensForNamenodes(), and causes the exception
        to happen. 
         org.apache.hadoop.mapreduce.security.TokenCache#obtainTokensForNamenodes
         public static void obtainTokensForNamenodes(Credentials credentials, Path[] ps, Configuration conf) throws IOException {
            if (!UserGroupInformation.isSecurityEnabled()) {
                return;
            }
            obtainTokensForNamenodesInternal(credentials, ps, conf);
        }	
        

        in hadoop1.2.1:
        In function "obtainTokensForNamenodesInternal",if path is in local mode, the fsName will be null. In this case, checks about whether the kerberos credentials exist or not will not be executed.
        org.apache.hadoop.mapreduce.security.TokenCache#obtainTokensForNamenodesInternal

         static void obtainTokensForNamenodesInternal(Credentials credentials,
                                                       Path [] ps, 
                                                       Configuration conf
                                                       ) throws IOException {
            // get jobtracker principal id (for the renewer)
            KerberosName jtKrbName = new KerberosName(conf.get(JobTracker.JT_USER_NAME, ""));
            String delegTokenRenewer = jtKrbName.getShortName();
            boolean readFile = true;
            for(Path p: ps) {
              FileSystem fs = FileSystem.get(p.toUri(), conf);
              String fsName = fs.getCanonicalServiceName();
              if (fsName == null) { 
                continue;
              }
              if (TokenCache.getDelegationToken(credentials, fsName) == null) {
                //TODO: Need to come up with a better place to put
                //this block of code to do with reading the file
                if (readFile) {
                  readFile = false;
                  String binaryTokenFilename =
                    conf.get(MAPREDUCE_JOB_CREDENTIALS_BINARY);
                  if (binaryTokenFilename != null) {
                    Credentials binary;
                    try {
                      binary = Credentials.readTokenStorageFile(new Path("file:///" +  
                          binaryTokenFilename), conf);
                    } catch (IOException e) {
                      throw new RuntimeException(e);
                    }
                    credentials.addAll(binary);
                  }
                  if (TokenCache.getDelegationToken(credentials, fsName) != null) {
                    LOG.debug("DT for " + fsName  + " is already present");
                    continue;
                  }
                }
                Token<?> token = fs.getDelegationToken(delegTokenRenewer);
                if (token != null) {
                  Text fsNameText = new Text(fsName);
                  credentials.addToken(fsNameText, token);
                  LOG.info("Got dt for " + p + ";uri="+ fsName + 
                           ";t.service="+token.getService());
                }
              }
            }
          }
          

        in hadoop2.3:
        In function "obtainTokensForNamenodesInternal", whether the path is in local mode or not, obtainTokensForNamenodesInternal always is executed. At that time, kerberos check fails in local mode.
        org.apache.hadoop.mapreduce.security.TokenCache#obtainTokensForNamenodesInternal

          static void obtainTokensForNamenodesInternal(Credentials credentials,
              Path[] ps, Configuration conf) throws IOException {
            Set<FileSystem> fsSet = new HashSet<FileSystem>();
            for(Path p: ps) {
              fsSet.add(p.getFileSystem(conf)); 
            }
            for (FileSystem fs : fsSet) {
              obtainTokensForNamenodesInternal(fs, credentials, conf);
            }
          }
          
          static void obtainTokensForNamenodesInternal(FileSystem fs, 
              Credentials credentials, Configuration conf) throws IOException {
            String delegTokenRenewer = Master.getMasterPrincipal(conf);
            if (delegTokenRenewer == null || delegTokenRenewer.length() == 0) {
              throw new IOException(
                  "Can't get Master Kerberos principal for use as renewer");
            }
            mergeBinaryTokens(credentials, conf);
        
            final Token<?> tokens[] = fs.addDelegationTokens(delegTokenRenewer,
                                                             credentials);
            if (tokens != null) {
              for (Token<?> token : tokens) {
                LOG.info("Got dt for " + fs.getUri() + "; "+token);
              }
            }
          }
        

        As Rohini Palaniswamy said:

        We should fix the bin/pig script to not load HADOOP_CONF_DIR in local mode.

        What patch did is not load the configurations in HADOOP_CONF_DIR in local mode.

        Show
        liyunzhang_intel added a comment - The problem that pig can not work in local mode with kerberos, only exists in hadoop2. As the root cause of the bug description said: When running pig in local mode, jobConf in HExecutionEngine is initiated with core-default.xml (hadoop.security.authentication = simple), mapred-default.xml, and yarn-default.xml. However, the settings are not passed to UserGroupInformation. That's why obtainTokensForNamenodesInternal() is called from obtainTokensForNamenodes(), and causes the exception to happen. org.apache.hadoop.mapreduce.security.TokenCache#obtainTokensForNamenodes public static void obtainTokensForNamenodes(Credentials credentials, Path[] ps, Configuration conf) throws IOException { if (!UserGroupInformation.isSecurityEnabled()) { return; } obtainTokensForNamenodesInternal(credentials, ps, conf); } in hadoop1.2.1: In function "obtainTokensForNamenodesInternal",if path is in local mode, the fsName will be null. In this case, checks about whether the kerberos credentials exist or not will not be executed. org.apache.hadoop.mapreduce.security.TokenCache#obtainTokensForNamenodesInternal static void obtainTokensForNamenodesInternal(Credentials credentials, Path [] ps, Configuration conf ) throws IOException { // get jobtracker principal id ( for the renewer) KerberosName jtKrbName = new KerberosName(conf.get(JobTracker.JT_USER_NAME, "")); String delegTokenRenewer = jtKrbName.getShortName(); boolean readFile = true ; for (Path p: ps) { FileSystem fs = FileSystem.get(p.toUri(), conf); String fsName = fs.getCanonicalServiceName(); if (fsName == null ) { continue ; } if (TokenCache.getDelegationToken(credentials, fsName) == null ) { //TODO: Need to come up with a better place to put // this block of code to do with reading the file if (readFile) { readFile = false ; String binaryTokenFilename = conf.get(MAPREDUCE_JOB_CREDENTIALS_BINARY); if (binaryTokenFilename != null ) { Credentials binary; try { binary = Credentials.readTokenStorageFile( new Path( "file: ///" + binaryTokenFilename), conf); } catch (IOException e) { throw new RuntimeException(e); } credentials.addAll(binary); } if (TokenCache.getDelegationToken(credentials, fsName) != null ) { LOG.debug( "DT for " + fsName + " is already present" ); continue ; } } Token<?> token = fs.getDelegationToken(delegTokenRenewer); if (token != null ) { Text fsNameText = new Text(fsName); credentials.addToken(fsNameText, token); LOG.info( "Got dt for " + p + ";uri=" + fsName + ";t.service=" +token.getService()); } } } } in hadoop2.3: In function "obtainTokensForNamenodesInternal", whether the path is in local mode or not, obtainTokensForNamenodesInternal always is executed. At that time, kerberos check fails in local mode. org.apache.hadoop.mapreduce.security.TokenCache#obtainTokensForNamenodesInternal static void obtainTokensForNamenodesInternal(Credentials credentials, Path[] ps, Configuration conf) throws IOException { Set<FileSystem> fsSet = new HashSet<FileSystem>(); for (Path p: ps) { fsSet.add(p.getFileSystem(conf)); } for (FileSystem fs : fsSet) { obtainTokensForNamenodesInternal(fs, credentials, conf); } } static void obtainTokensForNamenodesInternal(FileSystem fs, Credentials credentials, Configuration conf) throws IOException { String delegTokenRenewer = Master.getMasterPrincipal(conf); if (delegTokenRenewer == null || delegTokenRenewer.length() == 0) { throw new IOException( "Can't get Master Kerberos principal for use as renewer" ); } mergeBinaryTokens(credentials, conf); final Token<?> tokens[] = fs.addDelegationTokens(delegTokenRenewer, credentials); if (tokens != null ) { for (Token<?> token : tokens) { LOG.info( "Got dt for " + fs.getUri() + "; " +token); } } } As Rohini Palaniswamy said: We should fix the bin/pig script to not load HADOOP_CONF_DIR in local mode. What patch did is not load the configurations in HADOOP_CONF_DIR in local mode.
        Hide
        Xuefu Zhang added a comment -

        Hi Rohini Palaniswamy, Cheolsoo Park, I'm wondering if you have any additional thoughts on patch _1? Could we move this forward? Thanks.

        Show
        Xuefu Zhang added a comment - Hi Rohini Palaniswamy , Cheolsoo Park , I'm wondering if you have any additional thoughts on patch _1? Could we move this forward? Thanks.
        Hide
        Rohini Palaniswamy added a comment -

        Does not sound right to me. UserGroupInformation.java uses new Configuration(true) to initialize configuration.

        String value = conf.get(HADOOP_SECURITY_AUTHENTICATION);
            if (value == null || "simple".equals(value)) {
              useKerberos = false;
            }
        

        So unless there is some other configuration setting it to kerberos you should not be seeing the issue. Can you find out where it is being set to kerberos? Most likely it is being picked up from *-site.xml.

        Show
        Rohini Palaniswamy added a comment - Does not sound right to me. UserGroupInformation.java uses new Configuration(true) to initialize configuration. String value = conf.get(HADOOP_SECURITY_AUTHENTICATION); if (value == null || "simple" .equals(value)) { useKerberos = false ; } So unless there is some other configuration setting it to kerberos you should not be seeing the issue. Can you find out where it is being set to kerberos? Most likely it is being picked up from *-site.xml.
        Hide
        liyunzhang_intel added a comment -

        Hi Rohini Palaniswamy
        Thanks for your comment. Keberos security is widely used in hadoop. This bug was also found by other endusers(see http://www.ghostar.org/2014/05/pig-local-mode-fails-kerberos-auth-enabled/)
        detail steps to reproduce the bug:
        1. build kerbreos env
        2. build hadoop1 env
        which hadoop
        /home/zly/prj/oss/hadoop-1.2.1/bin/hadoop

        grep -C2 kerberos /home/zly/prj/oss/hadoop-1.2.1/conf/core-site.xml
        <name>hadoop.security.authentication</name>
        <value>kerberos</value>
        </property>

        3. build pig and run in local mode
        cd $PIG_HOME/bin
        ./pig -x local id.pig
        ps -ef|grep pig
        root 12126 10072 0 14:42 pts/4 00:00:01 /usr/java/jdk1.7.0_51//bin/java -Dproc_jar -Xmx1000m -Xmx1000m -Dpig.log.dir=/home/zly/prj/oss/pig/logs -Dpig.log.file=pig.log -Dpig.home.dir=/home/zly/prj/oss/pig -Xmx1000m -Dpig.log.dir=/home/zly/prj/oss/pig/logs -Dpig.log.file=pig.log -Dpig.home.dir=/home/zly/prj/oss/pig -Dhadoop.log.dir=/home/zly/prj/oss/hadoop-1.2.1/libexec/../logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/zly/prj/oss/hadoop-1.2.1/libexec/.. -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Dhadoop.security.logger=INFO,NullAppender -Djava.library.path=/home/zly/prj/oss/hadoop-1.2.1/libexec/../lib/native/Linux-amd64-64 -Dhadoop.policy.file=hadoop-policy.xml -Xrunjdwp:transport=dt_socket,server=y,address=9999 -classpath /home/zly/prj/oss/hadoop-1.2.1/conf:/home/zly/prj/oss/pig/conf:/usr/java/jdk1.7.0_51/lib/tools.jar:/home/zly/prj/oss/pig/build/ivy/lib/Pig/*:/home/zly/prj/oss/hadoop-1.2.1/conf:/home/zly/prj/oss/hadoop-1.2.1/conf:/home/zly/prj/oss/pig/lib/ST4-4.0.4.jar:/home/zly/prj/oss/pig/lib/accumulo-core-1.5.0.jar:......
        /home/zly/prj/oss/hadoop-1.2.1/conf:/home/zly/prj/oss/hadoop-1.2.1/conf is in the classpath

        Why UserGroupInformation set "hadoop.security.authentication" as "kerberos"

           UserGroupInformation#ensureInitialized
         private static synchronized void ensureInitialized() {
            if (!isInitialized) {
              initialize(new Configuration());
            }
          }
        
        UserGroupInformation#initialize
        private static synchronized void initialize(Configuration conf) {
            String value = conf.get(HADOOP_SECURITY_AUTHENTICATION);
            if (value == null || "simple".equals(value)) {
              useKerberos = false;
            } else if ("kerberos".equals(value)) {
              useKerberos = true;
            } else {
              throw new IllegalArgumentException("Invalid attribute value for " +
                                                 HADOOP_SECURITY_AUTHENTICATION + 
                                                 " of " + value);
            }
        

        The value of conf.get(HADOOP_SECURITY_AUTHENTICATION) decides "simple" or "kerberos".

        Referred https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/conf/Configuration.html
        Configurations are specified by resources. A resource contains a set of name/value pairs as XML data. Each resource is named by either a String or by a Path. If named by a String, then the classpath is examined for a file with that name. If named by a Path, then the local filesystem is examined directly, without referring to the classpath.
        Unless explicitly turned off, Hadoop by default specifies two resources, loaded in-order from the classpath:
        core-default.xml : Read-only defaults for hadoop.
        core-site.xml: Site-specific configuration for a given hadoop installation.
        core-default.xml in the jar and core-site.xml in the $HADOOP_HOME/conf/ are loaded. So the value of "hadoop.security.authentication" is "kerberos".

        If you don't build hadoop env and only deploy kerberos.
        1. build kerberos env
        2. build pig and run in local mode:
        cd $PIG_HOME/bin; ./pig -x local id.pig
        Error messages are found:
        exec /usr/java/jdk1.7.0_51/bin/java -Xmx1000m -Dpig.log.dir=/home/zly/prj/oss/pig/bin/../logs -Dpig.log.file=pig.log -Dpig.home.dir=/home/zly/prj/oss/pig/bin/.. -Xrunjdwp:transport=dt_socket,server=y,address=9999 -classpath /home/zly/prj/oss/pig/bin/../conf:/usr/java/jdk1.7.0_51/lib/tools.jar:/home/zly/prj/oss/pig/bin/../lib/ST4-4.0.4.jar:/home/zly/prj/oss/pig/bin/../lib/accumulo-core-1.5.0.jar:/home/zly/prj/oss/pig/bin/../lib/accumulo-fate-1.5.0.jar:/home/zly/prj/oss/pig/bin/../lib/accumulo-server-1.5.0.jar:/home/zly/prj/oss/pig/bin/../lib/accumulo-start-1.5.0.jar:/home/zly/prj/oss/pig/bin/../lib/accumulo-trace-1.5.0.jar:/home/zly/prj/oss/pig/bin/../lib/antlr-runtime-3.4.jar:/home/zly/prj/oss/pig/bin/../lib/asm-4.0.jar:/home/zly/prj/oss/pig/bin/../lib/asm-commons-4.0.jar:/home/zly/prj/oss/pig/bin/../lib/asm-tree-4.0.jar:/home/zly/prj/oss/pig/bin/../lib/automaton-1.11-8.jar:/home/zly/prj/oss/pig/bin/../lib/avro-1.7.5.jar:/home/zly/prj/oss/pig/bin/../lib/avro-tools-1.7.5-nodeps.jar:/home/zly/prj/oss/pig/bin/../lib/groovy-all-1.8.6.jar:/home/zly/prj/oss/pig/bin/../lib/guava-14.0.1.jar:/home/zly/prj/oss/pig/bin/../lib/hive-common-0.14.0-SNAPSHOT.jar:/home/zly/prj/oss/pig/bin/../lib/hive-exec-0.14.0-SNAPSHOT-core.jar:/home/zly/prj/oss/pig/bin/../lib/hive-serde-0.14.0-SNAPSHOT.jar:/home/zly/prj/oss/pig/bin/../lib/hive-shims-common-0.14.0-SNAPSHOT.jar:/home/zly/prj/oss/pig/bin/../lib/hive-shims-common-secure-0.14.0-SNAPSHOT.jar:/home/zly/prj/oss/pig/bin/../lib/jackson-core-asl-1.8.8.jar:/home/zly/prj/oss/pig/bin/../lib/jackson-mapper-asl-1.8.8.jar:/home/zly/prj/oss/pig/bin/../lib/jansi-1.9.jar:/home/zly/prj/oss/pig/bin/../lib/jline-1.0.jar:/home/zly/prj/oss/pig/bin/../lib/joda-time-2.1.jar:/home/zly/prj/oss/pig/bin/../lib/jruby-complete-1.6.7.jar:/home/zly/prj/oss/pig/bin/../lib/js-1.7R2.jar:/home/zly/prj/oss/pig/bin/../lib/json-simple-1.1.jar:/home/zly/prj/oss/pig/bin/../lib/jython-standalone-2.5.3.jar:/home/zly/prj/oss/pig/bin/../lib/protobuf-java-2.4.1-shaded.jar:/home/zly/prj/oss/pig/bin/../lib/protobuf-java-2.4.1.jar:/home/zly/prj/oss/pig/bin/../lib/protobuf-java-2.5.0.jar:/home/zly/prj/oss/pig/bin/../lib/snappy-java-1.0.5.jar:/home/zly/prj/oss/pig/bin/../lib/trevni-avro-1.7.5.jar:/home/zly/prj/oss/pig/bin/../lib/trevni-core-1.7.5.jar:/home/zly/prj/oss/pig/bin/../lib/zookeeper-3.4.5.jar:/home/zly/prj/oss/pig/bin/../pig-0.14.0-SNAPSHOT-core-h1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/avro-mapred-1.7.5.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hbase-client-0.96.0-hadoop1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hbase-common-0.96.0-hadoop1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hbase-hadoop-compat-0.96.0-hadoop1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hbase-hadoop1-compat-0.96.0-hadoop1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hbase-protocol-0.96.0-hadoop1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hbase-server-0.96.0-hadoop1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hive-shims-0.20S-0.14.0-SNAPSHOT.jar org.apache.pig.Main -x local id.pig
        Listening for transport dt_socket at address: 9999
        Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
        at org.apache.pig.Main.<clinit>(Main.java:100)
        Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 1 more
        Exception in thread "Thread-0" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/LocalFileSystem
        at org.apache.pig.Main$1.run(Main.java:95)
        Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.LocalFileSystem
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 1 more

        Show
        liyunzhang_intel added a comment - Hi Rohini Palaniswamy Thanks for your comment. Keberos security is widely used in hadoop. This bug was also found by other endusers(see http://www.ghostar.org/2014/05/pig-local-mode-fails-kerberos-auth-enabled/ ) detail steps to reproduce the bug: 1. build kerbreos env 2. build hadoop1 env which hadoop /home/zly/prj/oss/hadoop-1.2.1/bin/hadoop grep -C2 kerberos /home/zly/prj/oss/hadoop-1.2.1/conf/core-site.xml <name>hadoop.security.authentication</name> <value>kerberos</value> </property> 3. build pig and run in local mode cd $PIG_HOME/bin ./pig -x local id.pig ps -ef|grep pig root 12126 10072 0 14:42 pts/4 00:00:01 /usr/java/jdk1.7.0_51//bin/java -Dproc_jar -Xmx1000m -Xmx1000m -Dpig.log.dir=/home/zly/prj/oss/pig/logs -Dpig.log.file=pig.log -Dpig.home.dir=/home/zly/prj/oss/pig -Xmx1000m -Dpig.log.dir=/home/zly/prj/oss/pig/logs -Dpig.log.file=pig.log -Dpig.home.dir=/home/zly/prj/oss/pig -Dhadoop.log.dir=/home/zly/prj/oss/hadoop-1.2.1/libexec/../logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/zly/prj/oss/hadoop-1.2.1/libexec/.. -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Dhadoop.security.logger=INFO,NullAppender -Djava.library.path=/home/zly/prj/oss/hadoop-1.2.1/libexec/../lib/native/Linux-amd64-64 -Dhadoop.policy.file=hadoop-policy.xml -Xrunjdwp:transport=dt_socket,server=y,address=9999 -classpath /home/zly/prj/oss/hadoop-1.2.1/conf:/home/zly/prj/oss/pig/conf:/usr/java/jdk1.7.0_51/lib/tools.jar:/home/zly/prj/oss/pig/build/ivy/lib/Pig/*:/home/zly/prj/oss/hadoop-1.2.1/conf:/home/zly/prj/oss/hadoop-1.2.1/conf:/home/zly/prj/oss/pig/lib/ST4-4.0.4.jar:/home/zly/prj/oss/pig/lib/accumulo-core-1.5.0.jar:...... /home/zly/prj/oss/hadoop-1.2.1/conf:/home/zly/prj/oss/hadoop-1.2.1/conf is in the classpath Why UserGroupInformation set "hadoop.security.authentication" as "kerberos" UserGroupInformation#ensureInitialized private static synchronized void ensureInitialized() { if (!isInitialized) { initialize( new Configuration()); } } UserGroupInformation#initialize private static synchronized void initialize(Configuration conf) { String value = conf.get(HADOOP_SECURITY_AUTHENTICATION); if (value == null || "simple" .equals(value)) { useKerberos = false ; } else if ( "kerberos" .equals(value)) { useKerberos = true ; } else { throw new IllegalArgumentException( "Invalid attribute value for " + HADOOP_SECURITY_AUTHENTICATION + " of " + value); } The value of conf.get(HADOOP_SECURITY_AUTHENTICATION) decides "simple" or "kerberos". Referred https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/conf/Configuration.html Configurations are specified by resources. A resource contains a set of name/value pairs as XML data. Each resource is named by either a String or by a Path. If named by a String, then the classpath is examined for a file with that name. If named by a Path, then the local filesystem is examined directly, without referring to the classpath. Unless explicitly turned off, Hadoop by default specifies two resources, loaded in-order from the classpath: core-default.xml : Read-only defaults for hadoop. core-site.xml: Site-specific configuration for a given hadoop installation. core-default.xml in the jar and core-site.xml in the $HADOOP_HOME/conf/ are loaded. So the value of "hadoop.security.authentication" is "kerberos". If you don't build hadoop env and only deploy kerberos. 1. build kerberos env 2. build pig and run in local mode: cd $PIG_HOME/bin; ./pig -x local id.pig Error messages are found: exec /usr/java/jdk1.7.0_51/bin/java -Xmx1000m -Dpig.log.dir=/home/zly/prj/oss/pig/bin/../logs -Dpig.log.file=pig.log -Dpig.home.dir=/home/zly/prj/oss/pig/bin/.. -Xrunjdwp:transport=dt_socket,server=y,address=9999 -classpath /home/zly/prj/oss/pig/bin/../conf:/usr/java/jdk1.7.0_51/lib/tools.jar:/home/zly/prj/oss/pig/bin/../lib/ST4-4.0.4.jar:/home/zly/prj/oss/pig/bin/../lib/accumulo-core-1.5.0.jar:/home/zly/prj/oss/pig/bin/../lib/accumulo-fate-1.5.0.jar:/home/zly/prj/oss/pig/bin/../lib/accumulo-server-1.5.0.jar:/home/zly/prj/oss/pig/bin/../lib/accumulo-start-1.5.0.jar:/home/zly/prj/oss/pig/bin/../lib/accumulo-trace-1.5.0.jar:/home/zly/prj/oss/pig/bin/../lib/antlr-runtime-3.4.jar:/home/zly/prj/oss/pig/bin/../lib/asm-4.0.jar:/home/zly/prj/oss/pig/bin/../lib/asm-commons-4.0.jar:/home/zly/prj/oss/pig/bin/../lib/asm-tree-4.0.jar:/home/zly/prj/oss/pig/bin/../lib/automaton-1.11-8.jar:/home/zly/prj/oss/pig/bin/../lib/avro-1.7.5.jar:/home/zly/prj/oss/pig/bin/../lib/avro-tools-1.7.5-nodeps.jar:/home/zly/prj/oss/pig/bin/../lib/groovy-all-1.8.6.jar:/home/zly/prj/oss/pig/bin/../lib/guava-14.0.1.jar:/home/zly/prj/oss/pig/bin/../lib/hive-common-0.14.0-SNAPSHOT.jar:/home/zly/prj/oss/pig/bin/../lib/hive-exec-0.14.0-SNAPSHOT-core.jar:/home/zly/prj/oss/pig/bin/../lib/hive-serde-0.14.0-SNAPSHOT.jar:/home/zly/prj/oss/pig/bin/../lib/hive-shims-common-0.14.0-SNAPSHOT.jar:/home/zly/prj/oss/pig/bin/../lib/hive-shims-common-secure-0.14.0-SNAPSHOT.jar:/home/zly/prj/oss/pig/bin/../lib/jackson-core-asl-1.8.8.jar:/home/zly/prj/oss/pig/bin/../lib/jackson-mapper-asl-1.8.8.jar:/home/zly/prj/oss/pig/bin/../lib/jansi-1.9.jar:/home/zly/prj/oss/pig/bin/../lib/jline-1.0.jar:/home/zly/prj/oss/pig/bin/../lib/joda-time-2.1.jar:/home/zly/prj/oss/pig/bin/../lib/jruby-complete-1.6.7.jar:/home/zly/prj/oss/pig/bin/../lib/js-1.7R2.jar:/home/zly/prj/oss/pig/bin/../lib/json-simple-1.1.jar:/home/zly/prj/oss/pig/bin/../lib/jython-standalone-2.5.3.jar:/home/zly/prj/oss/pig/bin/../lib/protobuf-java-2.4.1-shaded.jar:/home/zly/prj/oss/pig/bin/../lib/protobuf-java-2.4.1.jar:/home/zly/prj/oss/pig/bin/../lib/protobuf-java-2.5.0.jar:/home/zly/prj/oss/pig/bin/../lib/snappy-java-1.0.5.jar:/home/zly/prj/oss/pig/bin/../lib/trevni-avro-1.7.5.jar:/home/zly/prj/oss/pig/bin/../lib/trevni-core-1.7.5.jar:/home/zly/prj/oss/pig/bin/../lib/zookeeper-3.4.5.jar:/home/zly/prj/oss/pig/bin/../pig-0.14.0-SNAPSHOT-core-h1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/avro-mapred-1.7.5.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hbase-client-0.96.0-hadoop1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hbase-common-0.96.0-hadoop1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hbase-hadoop-compat-0.96.0-hadoop1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hbase-hadoop1-compat-0.96.0-hadoop1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hbase-protocol-0.96.0-hadoop1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hbase-server-0.96.0-hadoop1.jar:/home/zly/prj/oss/pig/bin/../lib/h1/hive-shims-0.20S-0.14.0-SNAPSHOT.jar org.apache.pig.Main -x local id.pig Listening for transport dt_socket at address: 9999 Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory at org.apache.pig.Main.<clinit>(Main.java:100) Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 1 more Exception in thread "Thread-0" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/LocalFileSystem at org.apache.pig.Main$1.run(Main.java:95) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.LocalFileSystem at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 1 more
        Hide
        Rohini Palaniswamy added a comment - - edited

        Sorry. My bad. Forgot that core-site.xml is also loaded as part of defaults. We use a perl script instead of the bash bin/pig and what we do there is not add HADOOP_CONF_DIR to classpath if it is local mode.

        if [ "$HADOOP_CONF_DIR" != "" ]; then
            CLASSPATH=${CLASSPATH}:${HADOOP_CONF_DIR}
        fi
        

        should not be done if local mode. Either you can do that or even the current patch is ok. But you can simplify the code as below as you don't have to initialize UserGroupInformation in case of non local mode.

                            if (opts.getValStr().toLowerCase().contains("local") {
                                UserGroupInformation.setConfiguration(new Configuration(false));
                           }
        

        Above check also ensures kerberos is not enabled for tez_local mode. With 0.14 we are dropping support for Hadoop 0.20 so using UserGroupInformation class directly is ok at last ...

        Show
        Rohini Palaniswamy added a comment - - edited Sorry. My bad. Forgot that core-site.xml is also loaded as part of defaults. We use a perl script instead of the bash bin/pig and what we do there is not add HADOOP_CONF_DIR to classpath if it is local mode. if [ "$HADOOP_CONF_DIR" != "" ]; then CLASSPATH=${CLASSPATH}:${HADOOP_CONF_DIR} fi should not be done if local mode. Either you can do that or even the current patch is ok. But you can simplify the code as below as you don't have to initialize UserGroupInformation in case of non local mode. if (opts.getValStr().toLowerCase().contains( "local" ) { UserGroupInformation.setConfiguration( new Configuration( false )); } Above check also ensures kerberos is not enabled for tez_local mode. With 0.14 we are dropping support for Hadoop 0.20 so using UserGroupInformation class directly is ok at last ...
        Hide
        liyunzhang_intel added a comment -

        Hi Rohini Palaniswamy:
        Thanks for your comment. Patch has been updated and submitted. Can you help to review again?

        Show
        liyunzhang_intel added a comment - Hi Rohini Palaniswamy : Thanks for your comment. Patch has been updated and submitted. Can you help to review again?
        Hide
        Rohini Palaniswamy added a comment -

        +1. Committed to trunk (0.14). Thanks Liyun Zhang for the patch.

        Show
        Rohini Palaniswamy added a comment - +1. Committed to trunk (0.14). Thanks Liyun Zhang for the patch.

          People

          • Assignee:
            liyunzhang_intel
            Reporter:
            chiyang
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development