Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12279

Requesting a HBase table with kerberos is not working

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 1.5.2, 1.6.0
    • Fix Version/s: None
    • Component/s: YARN
    • Labels:
      None
    • Environment:

      Spark 1.6.0 / HBase 1.1.2 / Hadoop 2.7.1 / Zookeeper 3.4.5 / Authentication done through Kerberos

      Description

      I can't read a HBase table with Spark 1.5.2.

      I added the option "spark.driver.extraClassPath" in the spark-defaults.conf which contains the HBASE_CONF_DIR as below:
      spark.driver.extraClassPath = /opt/application/Hbase/current/conf/

      On the driver, I started spark-shell (I was running it in yarn-client mode)

      [my_user@uabigspark01 ~]$ spark-shell -v --name HBaseTest --jars /opt/application/Hbase/current/lib/hbase-common-1.1.2.jar,/opt/application/Hbase/current/lib/hbase-server-1.1.2.jar,/opt/application/Hbase/current/lib/hbase-client-1.1.2.jar,/opt/application/Hbase/current/lib/hbase-protocol-1.1.2.jar,/opt/application/Hbase/current/lib/protobuf-java-2.5.0.jar,/opt/application/Hbase/current/lib/htrace-core-3.1.0-incubating.jar,/opt/application/Hbase/current/lib/hbase-annotations-1.1.2.jar,/opt/application/Hbase/current/lib/guava-12.0.1.jar
      

      Then I ran the following lines:

      scala> import org.apache.spark._
      import org.apache.spark._
      
      scala> import org.apache.spark.rdd.NewHadoopRDD
      import org.apache.spark.rdd.NewHadoopRDD
      
      scala> import org.apache.hadoop.fs.Path
      import org.apache.hadoop.fs.Path
      
      scala> import org.apache.hadoop.hbase.util.Bytes
      import org.apache.hadoop.hbase.util.Bytes
      
      scala> import org.apache.hadoop.hbase.HColumnDescriptor
      import org.apache.hadoop.hbase.HColumnDescriptor
      
      scala> import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor}
      import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor}
      
      scala> import org.apache.hadoop.hbase.client.{HBaseAdmin, Put, HTable, Result}
      import org.apache.hadoop.hbase.client.{HBaseAdmin, Put, HTable, Result}
      
      scala> import org.apache.hadoop.hbase.mapreduce.TableInputFormat
      import org.apache.hadoop.hbase.mapreduce.TableInputFormat
      
      scala> import org.apache.hadoop.hbase.io.ImmutableBytesWritable
      import org.apache.hadoop.hbase.io.ImmutableBytesWritable
      
      scala> val conf = HBaseConfiguration.create()
      conf: org.apache.hadoop.conf.Configuration = Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, hbase-default.xml, hbase-site.xml
      
      scala> conf.addResource(new Path("/opt/application/Hbase/current/conf/hbase-site.xml"))
      
      scala> conf.set("hbase.zookeeper.quorum", "FQDN1:2181,FQDN2:2181,FQDN3:2181")
      
      scala> conf.set(TableInputFormat.INPUT_TABLE, "user:noheader")
      
      scala> val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat], classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable], classOf[org.apache.hadoop.hbase.client.Result])
      2015-12-09 15:17:58,890 INFO  [main] storage.MemoryStore: ensureFreeSpace(266248) called with curMem=0, maxMem=556038881
      2015-12-09 15:17:58,892 INFO  [main] storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 260.0 KB, free 530.0 MB)
      2015-12-09 15:17:59,196 INFO  [main] storage.MemoryStore: ensureFreeSpace(32808) called with curMem=266248, maxMem=556038881
      2015-12-09 15:17:59,197 INFO  [main] storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 32.0 KB, free 530.0 MB)
      2015-12-09 15:17:59,199 INFO  [sparkDriver-akka.actor.default-dispatcher-2] storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.200.208:60217 (size: 32.0 KB, free: 530.2 MB)
      2015-12-09 15:17:59,203 INFO  [main] spark.SparkContext: Created broadcast 0 from newAPIHadoopRDD at <console>:34
      hBaseRDD: org.apache.spark.rdd.RDD[(org.apache.hadoop.hbase.io.ImmutableBytesWritable, org.apache.hadoop.hbase.client.Result)] = NewHadoopRDD[0] at newAPIHadoopRDD at <console>:34
      
      scala> hBaseRDD.count()
      2015-12-09 15:18:09,441 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x4a52ac6 connecting to ZooKeeper ensemble=FQDN1:2181,FQDN2:2181,FQDN3:2181
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:host.name=DRIVER.FQDN
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:java.version=1.7.0_85
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:java.class.path=/opt/application/Hbase/current/conf/:/opt/application/Spark/current/conf/:/opt/application/Spark/current/assembly/target/scala-2.10/spark-assembly-1.5.2-hadoop2.7.1.jar:/opt/application/Spark/current/lib_managed/jars/datanucleus-core-3.2.10.jar:/opt/application/Spark/current/lib_managed/jars/datanucleus-rdbms-3.2.9.jar:/opt/application/Spark/current/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar:/opt/application/Hadoop/current/etc/hadoop/:/opt/application/Hadoop/current/etc/hadoop/
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.3.1.el6.x86_64
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=my_user
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/home/my_user
      2015-12-09 15:18:09,456 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/my_user
      2015-12-09 15:18:09,457 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=FQDN1:2181,FQDN2:2181,FQDN3:2181 sessionTimeout=90000 watcher=hconnection-0x4a52ac60x0, quorum=FQDN1:2181,FQDN2:2181,FQDN3:2181, baseZNode=/hbase
      2015-12-09 15:18:09,523 INFO  [main-SendThread(FQDN2:2181)] zookeeper.ClientCnxn: Opening socket connection to server FQDN2/192.168.200.24:2181. Will not attempt to authenticate using SASL (unknown error)
      2015-12-09 15:18:09,526 INFO  [main-SendThread(FQDN2:2181)] zookeeper.ClientCnxn: Socket connection established to FQDN2/192.168.200.24:2181, initiating session
      2015-12-09 15:18:09,540 INFO  [main-SendThread(FQDN2:2181)] zookeeper.ClientCnxn: Session establishment complete on server FQDN2/192.168.200.24:2181, sessionid = 0x25181db91c90010, negotiated timeout = 40000
      2015-12-09 15:18:09,685 INFO  [main] util.RegionSizeCalculator: Calculating region sizes for table "user:noheader".
      2015-12-09 15:18:10,192 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
      2015-12-09 15:18:10,192 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x25181db91c90010
      2015-12-09 15:18:10,196 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
      2015-12-09 15:18:10,196 INFO  [main] zookeeper.ZooKeeper: Session: 0x25181db91c90010 closed
      2015-12-09 15:18:10,335 INFO  [main] spark.SparkContext: Starting job: count at <console>:37
      2015-12-09 15:18:10,351 INFO  [dag-scheduler-event-loop] scheduler.DAGScheduler: Got job 0 (count at <console>:37) with 1 output partitions
      2015-12-09 15:18:10,351 INFO  [dag-scheduler-event-loop] scheduler.DAGScheduler: Final stage: ResultStage 0(count at <console>:37)
      2015-12-09 15:18:10,352 INFO  [dag-scheduler-event-loop] scheduler.DAGScheduler: Parents of final stage: List()
      2015-12-09 15:18:10,353 INFO  [dag-scheduler-event-loop] scheduler.DAGScheduler: Missing parents: List()
      2015-12-09 15:18:10,361 INFO  [dag-scheduler-event-loop] scheduler.DAGScheduler: Submitting ResultStage 0 (NewHadoopRDD[0] at newAPIHadoopRDD at <console>:34), which has no missing parents
      2015-12-09 15:18:10,402 INFO  [dag-scheduler-event-loop] storage.MemoryStore: ensureFreeSpace(1688) called with curMem=299056, maxMem=556038881
      2015-12-09 15:18:10,403 INFO  [dag-scheduler-event-loop] storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 1688.0 B, free 530.0 MB)
      2015-12-09 15:18:10,412 INFO  [dag-scheduler-event-loop] storage.MemoryStore: ensureFreeSpace(1067) called with curMem=300744, maxMem=556038881
      2015-12-09 15:18:10,412 INFO  [dag-scheduler-event-loop] storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1067.0 B, free 530.0 MB)
      2015-12-09 15:18:10,413 INFO  [sparkDriver-akka.actor.default-dispatcher-2] storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.200.208:60217 (size: 1067.0 B, free: 530.2 MB)
      2015-12-09 15:18:10,414 INFO  [dag-scheduler-event-loop] spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:861
      2015-12-09 15:18:10,418 INFO  [dag-scheduler-event-loop] scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (NewHadoopRDD[0] at newAPIHadoopRDD at <console>:34)
      2015-12-09 15:18:10,420 INFO  [dag-scheduler-event-loop] cluster.YarnScheduler: Adding task set 0.0 with 1 tasks
      2015-12-09 15:18:10,486 INFO  [sparkDriver-akka.actor.default-dispatcher-2] scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, EXECUTOR1.FQDN, RACK_LOCAL, 2716 bytes)
      2015-12-09 15:18:12,336 INFO  [sparkDriver-akka.actor.default-dispatcher-2] storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on EXECUTOR1.FQDN:44528 (size: 1067.0 B, free: 1589.8 MB)
      2015-12-09 15:18:12,888 INFO  [sparkDriver-akka.actor.default-dispatcher-2] storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on EXECUTOR1.FQDN:44528 (size: 32.0 KB, free: 1589.7 MB)
      2015-12-09 15:54:48,806 WARN  [task-result-getter-0] spark.ThrowableSerializationWrapper: Task exception could not be deserialized
      java.lang.ClassNotFoundException: org.apache.hadoop.hbase.client.RetriesExhaustedException
              at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
              at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
              at java.security.AccessController.doPrivileged(Native Method)
              at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
              at java.lang.Class.forName0(Native Method)
              at java.lang.Class.forName(Class.java:278)
              at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:67)
              at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
              at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
              at org.apache.spark.ThrowableSerializationWrapper.readObject(TaskEndReason.scala:167)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1897)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
              at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1997)
              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1921)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
              at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1997)
              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1921)
              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
              at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:72)
              at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:98)
              at org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$2.apply$mcV$sp(TaskResultGetter.scala:108)
              at org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$2.apply(TaskResultGetter.scala:105)
              at org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$2.apply(TaskResultGetter.scala:105)
              at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699)
              at org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(TaskResultGetter.scala:105)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      2015-12-09 15:54:48,813 ERROR [task-result-getter-0] scheduler.TaskResultGetter: Could not deserialize TaskEndReason: ClassNotFound with classloader org.apache.spark.repl.SparkIMain$TranslatingClassLoader@6735e614
      2015-12-09 15:54:48,814 WARN  [task-result-getter-0] scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, EXECUTOR1.FQDN): UnknownReason
      2015-12-09 15:54:48,819 INFO  [sparkDriver-akka.actor.default-dispatcher-16] scheduler.TaskSetManager: Starting task 0.1 in stage 0.0 (TID 1, EXECUTOR2.FQDN, RACK_LOCAL, 2716 bytes)
      2015-12-09 15:54:50,546 INFO  [sparkDriver-akka.actor.default-dispatcher-16] storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on EXECUTOR2.FQDN:45968 (size: 1067.0 B, free: 1589.8 MB)
      2015-12-09 15:54:51,128 INFO  [sparkDriver-akka.actor.default-dispatcher-16] storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on EXECUTOR2.FQDN:45968 (size: 32.0 KB, free: 1589.7 MB)
      

      During the execution of the "hBaseRDD.count()", the process was regularly paused (check the dates).
      At the same i saw the following errors on an executor (check the dates again).

      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/mnt/hd4/hadoop/yarn/local/usercache/my_user/filecache/766/spark-assembly-1.5.2-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/opt/application/Hadoop/hadoop-2.7.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      15/12/09 15:17:04 INFO executor.CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]
      15/12/09 15:17:05 INFO spark.SecurityManager: Changing view acls to: my_user
      15/12/09 15:17:05 INFO spark.SecurityManager: Changing modify acls to: my_user
      15/12/09 15:17:05 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(my_user); users with modify permissions: Set(my_user)
      15/12/09 15:17:05 INFO slf4j.Slf4jLogger: Slf4jLogger started
      15/12/09 15:17:05 INFO Remoting: Starting remoting
      15/12/09 15:17:05 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://driverPropsFetcher@EXECUTOR1.FQDN:36241]
      15/12/09 15:17:05 INFO util.Utils: Successfully started service 'driverPropsFetcher' on port 36241.
      15/12/09 15:17:06 INFO spark.SecurityManager: Changing view acls to: my_user
      15/12/09 15:17:06 INFO spark.SecurityManager: Changing modify acls to: my_user
      15/12/09 15:17:06 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(my_user); users with modify permissions: Set(my_user)
      15/12/09 15:17:06 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
      15/12/09 15:17:06 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
      15/12/09 15:17:06 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
      15/12/09 15:17:06 INFO slf4j.Slf4jLogger: Slf4jLogger started
      15/12/09 15:17:06 INFO Remoting: Starting remoting
      15/12/09 15:17:06 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutor@EXECUTOR1.FQDN:56581]
      15/12/09 15:17:06 INFO util.Utils: Successfully started service 'sparkExecutor' on port 56581.
      15/12/09 15:17:07 INFO storage.DiskBlockManager: Created local directory at /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/blockmgr-20cd4d44-6f07-4ea8-ab3d-6341e1bd94cd
      15/12/09 15:17:07 INFO storage.DiskBlockManager: Created local directory at /mnt/hd1/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/blockmgr-9a447538-ec8b-4803-a683-a7527ed9e00f
      15/12/09 15:17:07 INFO storage.DiskBlockManager: Created local directory at /mnt/hd2/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/blockmgr-bbd7b671-30a5-49eb-9a05-4fdfa1d86de7
      15/12/09 15:17:07 INFO storage.DiskBlockManager: Created local directory at /mnt/hd3/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/blockmgr-a4987072-5a10-4b96-a2bc-79477f5a8bd0
      15/12/09 15:17:07 INFO storage.DiskBlockManager: Created local directory at /mnt/hd4/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/blockmgr-3c85fdac-d29c-43b5-bcca-2296274f6381
      15/12/09 15:17:07 INFO storage.DiskBlockManager: Created local directory at /mnt/hd5/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/blockmgr-c8bb3422-f9d7-4e0b-8bb7-39c844d04bba
      15/12/09 15:17:07 INFO storage.DiskBlockManager: Created local directory at /mnt/hd6/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/blockmgr-e5de8172-1476-46ba-867a-288a5436efb9
      15/12/09 15:17:07 INFO storage.DiskBlockManager: Created local directory at /mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/blockmgr-fee9b2e2-2e03-4855-8f2f-44bd43006e01
      15/12/09 15:17:07 INFO storage.DiskBlockManager: Created local directory at /mnt/hd8/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/blockmgr-69b1f7d3-d35e-40b4-8cb4-437fae50fcfa
      15/12/09 15:17:07 INFO storage.DiskBlockManager: Created local directory at /mnt/hd9/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/blockmgr-11b53137-59ba-4e16-8faa-1f2a4c75c721
      15/12/09 15:17:07 INFO storage.DiskBlockManager: Created local directory at /mnt/hd10/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/blockmgr-ade507aa-9b4c-44b6-b27d-1485cd7dd66a
      15/12/09 15:17:07 INFO storage.DiskBlockManager: Created local directory at /mnt/hd11/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/blockmgr-443240c7-895a-4241-9e4d-89de48e0bca1
      15/12/09 15:17:07 INFO storage.MemoryStore: MemoryStore started with capacity 1589.8 MB
      15/12/09 15:17:08 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://sparkDriver@DRIVER.FQDN:62433/user/CoarseGrainedScheduler
      15/12/09 15:17:08 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with driver
      15/12/09 15:17:08 INFO executor.Executor: Starting executor ID 2 on host EXECUTOR1.FQDN
      15/12/09 15:17:08 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44528.
      15/12/09 15:17:08 INFO netty.NettyBlockTransferService: Server created on 44528
      15/12/09 15:17:08 INFO storage.BlockManagerMaster: Trying to register BlockManager
      15/12/09 15:17:08 INFO storage.BlockManagerMaster: Registered BlockManager
      15/12/09 15:17:08 INFO executor.Executor: Using REPL class URI: http://DRIVER.FQDN:53441
      15/12/09 15:18:10 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 0
      15/12/09 15:18:10 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
      15/12/09 15:18:10 INFO executor.Executor: Fetching http://DRIVER.FQDN:58525/jars/protobuf-java-2.5.0.jar with timestamp 1449670596931
      15/12/09 15:18:10 INFO util.Utils: Fetching http://DRIVER.FQDN:58525/jars/protobuf-java-2.5.0.jar to /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/fetchFileTemp154889812017775736.tmp
      15/12/09 15:18:10 INFO util.Utils: Copying /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/21289125271449670596931_cache to /mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./protobuf-java-2.5.0.jar
      15/12/09 15:18:10 INFO executor.Executor: Adding file:/mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./protobuf-java-2.5.0.jar to class loader
      15/12/09 15:18:10 INFO executor.Executor: Fetching http://DRIVER.FQDN:58525/jars/hbase-common-1.1.2.jar with timestamp 1449670596905
      15/12/09 15:18:10 INFO util.Utils: Fetching http://DRIVER.FQDN:58525/jars/hbase-common-1.1.2.jar to /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/fetchFileTemp3207469898154433839.tmp
      15/12/09 15:18:10 INFO util.Utils: Copying /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/17557465671449670596905_cache to /mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./hbase-common-1.1.2.jar
      15/12/09 15:18:10 INFO executor.Executor: Adding file:/mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./hbase-common-1.1.2.jar to class loader
      15/12/09 15:18:10 INFO executor.Executor: Fetching http://DRIVER.FQDN:58525/jars/hbase-server-1.1.2.jar with timestamp 1449670596916
      15/12/09 15:18:10 INFO util.Utils: Fetching http://DRIVER.FQDN:58525/jars/hbase-server-1.1.2.jar to /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/fetchFileTemp1412976626726073257.tmp
      15/12/09 15:18:10 INFO util.Utils: Copying /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/-7756328011449670596916_cache to /mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./hbase-server-1.1.2.jar
      15/12/09 15:18:10 INFO executor.Executor: Adding file:/mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./hbase-server-1.1.2.jar to class loader
      15/12/09 15:18:10 INFO executor.Executor: Fetching http://DRIVER.FQDN:58525/jars/hbase-annotations-1.1.2.jar with timestamp 1449670596935
      15/12/09 15:18:10 INFO util.Utils: Fetching http://DRIVER.FQDN:58525/jars/hbase-annotations-1.1.2.jar to /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/fetchFileTemp2679836512883108007.tmp
      15/12/09 15:18:10 INFO util.Utils: Copying /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/4608020401449670596935_cache to /mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./hbase-annotations-1.1.2.jar
      15/12/09 15:18:10 INFO executor.Executor: Adding file:/mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./hbase-annotations-1.1.2.jar to class loader
      15/12/09 15:18:10 INFO executor.Executor: Fetching http://DRIVER.FQDN:58525/jars/htrace-core-3.1.0-incubating.jar with timestamp 1449670596935
      15/12/09 15:18:10 INFO util.Utils: Fetching http://DRIVER.FQDN:58525/jars/htrace-core-3.1.0-incubating.jar to /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/fetchFileTemp1461961544034139965.tmp
      15/12/09 15:18:10 INFO util.Utils: Copying /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/-10342791481449670596935_cache to /mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./htrace-core-3.1.0-incubating.jar
      15/12/09 15:18:10 INFO executor.Executor: Adding file:/mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./htrace-core-3.1.0-incubating.jar to class loader
      15/12/09 15:18:10 INFO executor.Executor: Fetching http://DRIVER.FQDN:58525/jars/hbase-client-1.1.2.jar with timestamp 1449670596919
      15/12/09 15:18:10 INFO util.Utils: Fetching http://DRIVER.FQDN:58525/jars/hbase-client-1.1.2.jar to /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/fetchFileTemp9217964731951772544.tmp
      15/12/09 15:18:10 INFO util.Utils: Copying /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/-7183065851449670596919_cache to /mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./hbase-client-1.1.2.jar
      15/12/09 15:18:10 INFO executor.Executor: Adding file:/mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./hbase-client-1.1.2.jar to class loader
      15/12/09 15:18:10 INFO executor.Executor: Fetching http://DRIVER.FQDN:58525/jars/hbase-protocol-1.1.2.jar with timestamp 1449670596930
      15/12/09 15:18:10 INFO util.Utils: Fetching http://DRIVER.FQDN:58525/jars/hbase-protocol-1.1.2.jar to /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/fetchFileTemp8280539781420543251.tmp
      15/12/09 15:18:10 INFO util.Utils: Copying /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/-5481020281449670596930_cache to /mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./hbase-protocol-1.1.2.jar
      15/12/09 15:18:10 INFO executor.Executor: Adding file:/mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./hbase-protocol-1.1.2.jar to class loader
      15/12/09 15:18:10 INFO executor.Executor: Fetching http://DRIVER.FQDN:58525/jars/guava-12.0.1.jar with timestamp 1449670596939
      15/12/09 15:18:10 INFO util.Utils: Fetching http://DRIVER.FQDN:58525/jars/guava-12.0.1.jar to /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/fetchFileTemp6766780549158760761.tmp
      15/12/09 15:18:10 INFO util.Utils: Copying /mnt/hd0/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/spark-58907fb7-5a3d-49af-b675-37a7cdd6ba27/-14937723981449670596939_cache to /mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./guava-12.0.1.jar
      15/12/09 15:18:10 INFO executor.Executor: Adding file:/mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/./guava-12.0.1.jar to class loader
      15/12/09 15:18:11 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 1
      15/12/09 15:18:12 INFO storage.MemoryStore: ensureFreeSpace(1067) called with curMem=0, maxMem=1666984181
      15/12/09 15:18:12 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1067.0 B, free 1589.8 MB)
      15/12/09 15:18:12 INFO broadcast.TorrentBroadcast: Reading broadcast variable 1 took 1214 ms
      15/12/09 15:18:12 INFO storage.MemoryStore: ensureFreeSpace(1688) called with curMem=1067, maxMem=1666984181
      15/12/09 15:18:12 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 1688.0 B, free 1589.8 MB)
      15/12/09 15:18:12 INFO rdd.NewHadoopRDD: Input split: HBase table split(table name: my_user:noheader, scan: , start row: , end row: , region location: EXECUTOR3.FQDN)
      15/12/09 15:18:12 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 0
      15/12/09 15:18:12 INFO storage.MemoryStore: ensureFreeSpace(32808) called with curMem=2755, maxMem=1666984181
      15/12/09 15:18:12 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 32.0 KB, free 1589.7 MB)
      15/12/09 15:18:12 INFO broadcast.TorrentBroadcast: Reading broadcast variable 0 took 25 ms
      15/12/09 15:18:13 INFO storage.MemoryStore: ensureFreeSpace(398840) called with curMem=35563, maxMem=1666984181
      15/12/09 15:18:13 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 389.5 KB, free 1589.3 MB)
      15/12/09 15:18:13 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x46aff06c connecting to ZooKeeper ensemble=FQDN1:2181,FQDN2:2181,FQDN3:2181
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:host.name=EXECUTOR1.FQDN
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_79
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/jre
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003:/mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/__spark__.jar:/opt/application/Hadoop/current/etc/hadoop/:/opt/application/Hadoop/current/share/hadoop/common/hadoop-nfs-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/common/hadoop-common-2.7.1-tests.jar:/opt/application/Hadoop/current/share/hadoop/common/hadoop-common-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/hadoop-auth-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/httpcore-4.2.5.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/avro-1.7.4.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-net-3.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/hadoop-annotations-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-io-2.4.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/paranamer-2.3.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/guava-11.0.2.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jettison-1.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jsch-0.1.42.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/activation-1.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/junit-4.11.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/zookeeper-3.4.6.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/asm-3.2.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/xz-1.0.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/gson-2.2.4.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/httpclient-4.2.5.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/application/Hadoop/current/share/hadoop/common/lib/commons-collections-3.2.1.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/hadoop-hdfs-2.7.1-tests.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/hadoop-hdfs-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/application/Hadoop/current/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/application/Hadoop/current/share/hadoop/yarn/hadoop-yarn-common-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/hadoop-yarn-client-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/hadoop-yarn-server-common-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/hadoop-yarn-api-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/hadoop-yarn-registry-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/guice-3.0.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/guava-11.0.2.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/activation-1.1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/asm-3.2.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/xz-1.0.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/application/Hadoop/current/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1-tests.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.1.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/javax.inject-1.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/application/Hadoop/current/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/application/Hadoop/current/lib/native/
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003/tmp
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-504.8.1.el6.x86_64
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:user.name=my_user
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/my_user
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Client environment:user.dir=/mnt/hd7/hadoop/yarn/local/usercache/my_user/appcache/application_1444912162557_0919/container_1444912162557_0919_01_000003
      15/12/09 15:18:13 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=FQDN1:2181,FQDN2:2181,FQDN3:2181 sessionTimeout=90000 watcher=hconnection-0x46aff06c0x0, quorum=FQDN1:2181,FQDN2:2181,FQDN3:2181, baseZNode=/hbase
      15/12/09 15:18:13 INFO zookeeper.ClientCnxn: Opening socket connection to server FQDN3/192.168.200.25:2181. Will not attempt to authenticate using SASL (unknown error)
      15/12/09 15:18:13 INFO zookeeper.ClientCnxn: Socket connection established to FQDN3/192.168.200.25:2181, initiating session
      15/12/09 15:18:13 INFO zookeeper.ClientCnxn: Session establishment complete on server FQDN3/192.168.200.25:2181, sessionid = 0x35181db91ef0010, negotiated timeout = 40000
      15/12/09 15:18:14 INFO mapreduce.TableInputFormatBase: Input split length: 0 bytes.
      15/12/09 15:18:14 WARN ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      15/12/09 15:18:14 ERROR ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
      	at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:642)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:166)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:769)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:766)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:766)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:920)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:889)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1222)
      	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
      	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
      	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
      	at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:201)
      	at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:180)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:346)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:320)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
      	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
      	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
      	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
      	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
      	... 25 more
      15/12/09 15:18:18 WARN ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      15/12/09 15:18:18 ERROR ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
      	at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:642)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:166)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:769)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:766)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:766)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:920)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:889)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1222)
      	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
      	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
      	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
      	at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:201)
      	at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:180)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:346)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:320)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
      	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
      	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
      	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
      	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
      	... 25 more
      	...
      MORE THAN ONE HUNDRED THOUSAND LINES LIKE THE PREVIOUS ONE...
      	...
      15/12/09 15:54:00 WARN ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      15/12/09 15:54:00 ERROR ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
      	at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:642)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:166)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:769)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:766)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:766)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:920)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:889)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1222)
      	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
      	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
      	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
      	at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:201)
      	at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:180)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:346)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:320)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
      	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
      	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
      	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
      	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
      	... 25 more
      15/12/09 15:54:04 WARN ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      15/12/09 15:54:04 ERROR ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
      	at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:642)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:166)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:769)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:766)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:766)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:920)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:889)
      	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1222)
      	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
      	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
      	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
      	at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:201)
      	at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:180)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:346)
      	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:320)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
      	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
      	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
      	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
      	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
      	... 25 more
      

        Issue Links

          Activity

          Hide
          ybodnar Y Bodnar added a comment -

          Got it working on Spark 1.5.2 (built for hadoop2.4) with Spark Streaming job:

          When spark-submit detects YARN cluster deployment mode, org.apache.spark.deploy.yarn.Client is used for app submission.
          It's method obtainTokenForHBase requires hbase.security.authentication to be set to "true" in configuration + set of HBase dependencies available to spark-submit application (not --driver-class-path but actually spark-submit itself).

          In order for functionality to work, I had to set following in $SPARK_CONF_DIR/spark-env.sh:

          • YARN_CONF_DIR
          • HADOOP_CONF_DIR
          • HADOOP_HOME
          • SPARK_CLASSPATH - this has to include following jars: htrace-core-2.04.jar:hbase-client.jar:hbase-common.jar:hbase-server.jar:guava-11.0.2.jar:hadoop-common.jar. These are in HBase Lib dir. Exact JAR names will differ depending on environment setup.

          Other important thing is to have hive-site.xml available for spark-submit. I just copied it into $SPARK_CONF_DIR.

          Now, despite the fact driver runs in Application Master when YARN cluster deployment mode is used, org.apache.spark.deploy.yarn.Client is doing actual submission from local machine on which spark-submit is executed. That's why spark-env.sh must be updated with all these jars despite the fact some of them might be included in your uber jar or provided via --jars parameter.

          For debugging purposes, do following:

          • In $SPARK_CONF_DIR/log4j.properties, update log4j.rootCategory=DEBUG, ... - this will make sure yarn.Client show messages related to obtainTokenForHBase.
          • in output of spark-submit command on local machine (not AM nor executor logs will show it) you should see message "Added HBase security token to credentials." - this is indicator that token has been obtained.
          • If message from point above is not there then look for "HBase Method not found", "HBase Class not found" or * "Exception when obtaining HBase security token: " messages. These will show up if jars hbase-client, hadoop-common or other jars are not in SPARK_CLASSPATH.
          Show
          ybodnar Y Bodnar added a comment - Got it working on Spark 1.5.2 (built for hadoop2.4) with Spark Streaming job: When spark-submit detects YARN cluster deployment mode, org.apache.spark.deploy.yarn.Client is used for app submission. It's method obtainTokenForHBase requires hbase.security.authentication to be set to "true" in configuration + set of HBase dependencies available to spark-submit application (not --driver-class-path but actually spark-submit itself). In order for functionality to work, I had to set following in $SPARK_CONF_DIR/spark-env.sh: YARN_CONF_DIR HADOOP_CONF_DIR HADOOP_HOME SPARK_CLASSPATH - this has to include following jars: htrace-core-2.04.jar:hbase-client.jar:hbase-common.jar:hbase-server.jar:guava-11.0.2.jar:hadoop-common.jar. These are in HBase Lib dir. Exact JAR names will differ depending on environment setup. Other important thing is to have hive-site.xml available for spark-submit. I just copied it into $SPARK_CONF_DIR. Now, despite the fact driver runs in Application Master when YARN cluster deployment mode is used, org.apache.spark.deploy.yarn.Client is doing actual submission from local machine on which spark-submit is executed. That's why spark-env.sh must be updated with all these jars despite the fact some of them might be included in your uber jar or provided via --jars parameter. For debugging purposes, do following: In $SPARK_CONF_DIR/log4j.properties, update log4j.rootCategory=DEBUG, ... - this will make sure yarn.Client show messages related to obtainTokenForHBase. in output of spark-submit command on local machine (not AM nor executor logs will show it) you should see message "Added HBase security token to credentials." - this is indicator that token has been obtained. If message from point above is not there then look for "HBase Method not found", "HBase Class not found" or * "Exception when obtaining HBase security token: " messages. These will show up if jars hbase-client, hadoop-common or other jars are not in SPARK_CLASSPATH.
          Hide
          pbeauvois Pierre Beauvois added a comment -

          Hi Y Bodnar, thanks for your Spark configuration. I tested this morning but unfortunately I wasn't able to make it work.

          Here is my spark-env.sh configuration:

          export HADOOP_HOME=/opt/application/Hadoop/current
          export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
          
          export YARN_CONF_DIR=/opt/application/Hadoop/current/etc/hadoop
          
          export SPARK_HOME=/opt/application/Spark/current
          export SPARK_CLASSPATH=${SPARK_HOME}/lib_managed/jars/hbase-common.jar:${SPARK_HOME}/lib_managed/jars/hbase-server.jar:${SPARK_HOME}/lib_managed/jars/hbase-client.jar:${SPARK_HOME}/lib_managed/jars/hbase-protocol.jar:${SPARK_HOME}/lib_managed/jars/protobuf-java.jar:${SPARK_HOME}/lib_managed/jars/htrace-core-incubating.jar:${SPARK_HOME}/lib_managed/jars/hbase-annotations.jar:${SPARK_HOME}/lib_managed/jars/guava.jar:${SPARK_HOME}/lib_managed/jars/hadoop-common.jar
          
          export HBASE_CONF_DIR=/opt/application/Hbase/current/conf
          

          You'll note that I added the HBase libraries from the Spark lib directory. This is not a mistake, I created symbolic links in order to make it easier to manage.

          In order to do the test, I used the code example which is available with Spark:

          [pbeauvois@uabigspark01 ~]$ spark-submit -v --master yarn-client --class org.apache.spark.examples.HBaseTest /opt/application/Spark/current/examples/target/spark-examples_2.10-1.5.2.jar pbeauvois:noheader
          

          The result here gives no errors related to something like "HBase security token" but a Zookeeper connection error is regularly displayed (it also appears if I add conf.set("hbase.zookeeper.quorum", "server1:2181,server2:2181,server3:2181")in the code):

          15/12/21 10:47:56 DEBUG SaslRpcClient: Sending sasl message state: INITIATE
          token: "`\202\002u\006\t*\206H\206\367\022\001\002\002\001\000n\202\002d0\202\002`\240\003\002\001\005\241\003\002\001\016\242\a\003\005\000 \000\000\000\243\202\001ra\202\001n0\202\001j\240\003\002\001\005\241\020\033\016SANDBOX.HADOOP\242301\240\003\002\001\000\241*0(\033\002nn\033\"uabigname02.rouen.francetelecom.fr\243\202\001\0320\202\001\026\240\003\002\001\022\241\003\002\001\a\242\202\001\b\004\202\001\004}\276$\220\240\2546\306\234\034\017W\354m\032\256\255\324\202\333\221\244\227\310bk\222\346\334\036\324\275a\262\360\221\315)\217\232Av\211\343\024\300\224\360\020ry\265\235\336\025\247\025\264\037B\3246\351\f\231\022\223h\254\277\241|dc\361\2037\305\2601\000\006\346\213qH\307\3452\r\306I\320\243\324\tu0b\367@\36\3727\224\214\002\\\317b2\217)\277\201X\356\244\370\310\246\004\304\362\262x\201\202\315\213\025{\361O\214*\216 \340\'\351@\331&\022\226\3727\240\233\201~\326\254\361\270*>z1\366G%98\0247\321\004\030?\242\203\3216\202%!z\336\002,\300\037\365\332\357\003\266\355RK\247,\026\271\274D\aC\335\222\304\313\203\'n\310z\350V\341\311\345\301\206\245/\224]\003\221\by]\325\305\333\314/P\277\035\032\000\273\266&\200n\225\352\375q\n\371\201\3704\366\000\344\'\352\352\244\201\3240\201\321\240\003\002\001\022\242\201\311\004\201\306\200\223\vd\273\252A\2259%\267\363\'\375\375u\000\346\307\377\346\276@\004I^\337\303d\232jv\347^[\034\276\355\260\260\355\263\016\027\035\033\034\327-\r\"7\254\377\214\211\370\331jk:\360:\312@\262\221<Ce~\337u,\214\351\327I\024`Q*\235\351\245\314\211\004\340\2261\251\032\220\336\230/o\'>\310\255Iu@g\323Wp=\360yc\332\033\367\226\264\240l\205\2471O/\020Dm\371\377\273$W\342\273\034\027\307\341\016\226\3179\236\000\253m\361\3522[\211fZ\242\037A\334\v\240\033\242\337\203\245;_\003\020s\376\3509z\220\275\277\266\204\232,(=#hE\350\303\346\264\027q\002([+\373\201"
          auths {
            method: "KERBEROS"
            mechanism: "GSSAPI"
            protocol: "nn"
            serverId: "server2"
          }
          
          15/12/21 10:47:56 DEBUG SaslRpcClient: Received SASL message state: CHALLENGE
          token: "`j\006\t*\206H\206\367\022\001\002\002\002\000o[0Y\240\003\002\001\005\241\003\002\001\017\242M0K\240\003\002\001\022\242D\004B\203\3422\031\250\024\265&\340\247\3161\254\022\300nTN\257\276\242<\211\361mw\353f_\344\374\017\306*B\265\005}\356\210\375\226\\\326\250\302\243\314\251\315]:\332Nx>\324\303;\317Y\332\317w"
          
          15/12/21 10:47:56 DEBUG SaslRpcClient: Sending sasl message state: RESPONSE
          token: ""
          
          15/12/21 10:47:56 DEBUG SaslRpcClient: Received SASL message state: CHALLENGE
          token: "\005\004\001\377\000\f\000\000\000\000\000\000\a2\025\223\001\001\000\000\025\254\334\363C\002\023\247h\272\270\240"
          
          15/12/21 10:47:56 DEBUG SaslRpcClient: Sending sasl message state: RESPONSE
          token: "\005\004\000\377\000\f\000\000\000\000\000\000\016\\3\020\001\001\000\0005\322\312\330;\355\367U}N"
          
          15/12/21 10:47:56 DEBUG SaslRpcClient: Received SASL message state: SUCCESS
          
          15/12/21 10:47:56 DEBUG Client: Negotiated QOP is :auth
          15/12/21 10:47:56 DEBUG Client: IPC Client (1212945869) connection to server2/192.168.200.24:8020 from pbeauvois@SANDBOX.HADOOP sending #59
          15/12/21 10:47:56 DEBUG Client: IPC Client (1212945869) connection to server2/192.168.200.24:8020 from pbeauvois@SANDBOX.HADOOP: starting, having connections 2
          15/12/21 10:47:56 DEBUG Client: IPC Client (1212945869) connection to server2/192.168.200.24:8020 from pbeauvois@SANDBOX.HADOOP got value #59
          15/12/21 10:47:56 DEBUG ProtobufRpcEngine: Call: renewLease took 30ms
          15/12/21 10:47:56 DEBUG LeaseRenewer: Lease renewed for client DFSClient_NONMAPREDUCE_-729245058_1
          15/12/21 10:47:56 DEBUG LeaseRenewer: Lease renewer daemon for [DFSClient_NONMAPREDUCE_-729245058_1] with renew id 1 executed
          15/12/21 10:47:56 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(ReviveOffers,false) from Actor[akka://sparkDriver/deadLetters]
          15/12/21 10:47:56 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(ReviveOffers,false)
          15/12/21 10:47:56 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.682931 ms) AkkaMessage(ReviveOffers,false) from Actor[akka://sparkDriver/deadLetters]
          15/12/21 10:48:01 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
          15/12/21 10:48:01 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
          java.net.ConnectException: Connection refused
                  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
                  at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
                  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
          15/12/21 10:48:01 DEBUG ClientCnxnSocketNIO: Ignoring exception during shutdown input
          java.nio.channels.ClosedChannelException
                  at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:807)
                  at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:426)
                  at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:189)
                  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
                  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)
          15/12/21 10:48:01 DEBUG ClientCnxnSocketNIO: Ignoring exception during shutdown output
          java.nio.channels.ClosedChannelException
                  at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:824)
                  at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:434)
                  at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:196)
                  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
                  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)
          15/12/21 10:48:01 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(ReviveOffers,false) from Actor[akka://sparkDriver/deadLetters]
          

          I tried to configure Spark differently. Setting SPARK_HOME and SPARK_CLASSPATH on spark-env.sh seem to be deprecated. So I configured spark-env.sh as the following:

          export HADOOP_HOME=/opt/application/Hadoop/current
          export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
          
          export YARN_CONF_DIR=/opt/application/Hadoop/current/etc/hadoop
          
          #export SPARK_HOME=/opt/application/Spark/current
          #export SPARK_CLASSPATH=${SPARK_HOME}/lib_managed/jars/hbase-common.jar:${SPARK_HOME}/lib_managed/jars/hbase-server.jar:${SPARK_HOME}/lib_managed/jars/hbase-client.jar:${SPARK_HOME}/lib_managed/jars/hbase-protocol.jar:${SPARK_HOME}/lib_managed/jars/protobuf-java.jar:${SPARK_HOME}/lib_managed/jars/htrace-core-incubating.jar:${SPARK_HOME}/lib_managed/jars/hbase-annotations.jar:${SPARK_HOME}/lib_managed/jars/guava.jar:${SPARK_HOME}/lib_managed/jars/hadoop-common.jar
          
          #export HBASE_CONF_DIR=/opt/application/Hbase/current/conf
          

          and spark-defaults.conf:

          spark.driver.extraClassPath = /opt/application/Hbase/current/conf/
          

          Then I launched spark-submit like that:

          [pbeauvois@uabigspark01 ~]$ spark-submit -v --master yarn-client --class org.apache.spark.examples.HBaseTest --jars /opt/application/Spark/current/lib_managed/jars/hbase-common.jar,/opt/application/Spark/current/lib_managed/jars/hbase-server.jar,/opt/application/Spark/current/lib_managed/jars/hbase-client.jar,/opt/application/Spark/current/lib_managed/jars/hbase-protocol.jar,/opt/application/Spark/current/lib_managed/jars/protobuf-java.jar,/opt/application/Spark/current/lib_managed/jars/htrace-core-incubating.jar,/opt/application/Spark/current/lib_managed/jars/hbase-annotations.jar,/opt/application/Spark/current/lib_managed/jars/guava.jar,/opt/application/Spark/current/lib_managed/jars/hadoop-common.jar /opt/application/Spark/current/examples/target/spark-examples_2.10-1.5.2.jar pbeauvois:noheader
          

          In that case, the error is the same as the one I posted in the description.

          As a conclusion, I can't use Spark to request HBase tables for the moment. Any ideas?

          Show
          pbeauvois Pierre Beauvois added a comment - Hi Y Bodnar , thanks for your Spark configuration. I tested this morning but unfortunately I wasn't able to make it work. Here is my spark-env.sh configuration: export HADOOP_HOME=/opt/application/Hadoop/current export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop export YARN_CONF_DIR=/opt/application/Hadoop/current/etc/hadoop export SPARK_HOME=/opt/application/Spark/current export SPARK_CLASSPATH=${SPARK_HOME}/lib_managed/jars/hbase-common.jar:${SPARK_HOME}/lib_managed/jars/hbase-server.jar:${SPARK_HOME}/lib_managed/jars/hbase-client.jar:${SPARK_HOME}/lib_managed/jars/hbase-protocol.jar:${SPARK_HOME}/lib_managed/jars/protobuf-java.jar:${SPARK_HOME}/lib_managed/jars/htrace-core-incubating.jar:${SPARK_HOME}/lib_managed/jars/hbase-annotations.jar:${SPARK_HOME}/lib_managed/jars/guava.jar:${SPARK_HOME}/lib_managed/jars/hadoop-common.jar export HBASE_CONF_DIR=/opt/application/Hbase/current/conf You'll note that I added the HBase libraries from the Spark lib directory. This is not a mistake, I created symbolic links in order to make it easier to manage. In order to do the test, I used the code example which is available with Spark: [pbeauvois@uabigspark01 ~]$ spark-submit -v --master yarn-client --class org.apache.spark.examples.HBaseTest /opt/application/Spark/current/examples/target/spark-examples_2.10-1.5.2.jar pbeauvois:noheader The result here gives no errors related to something like "HBase security token" but a Zookeeper connection error is regularly displayed (it also appears if I add conf.set("hbase.zookeeper.quorum", "server1:2181,server2:2181,server3:2181")in the code): 15/12/21 10:47:56 DEBUG SaslRpcClient: Sending sasl message state: INITIATE token: "`\202\002u\006\t*\206H\206\367\022\001\002\002\001\000n\202\002d0\202\002`\240\003\002\001\005\241\003\002\001\016\242\a\003\005\000 \000\000\000\243\202\001ra\202\001n0\202\001j\240\003\002\001\005\241\020\033\016SANDBOX.HADOOP\242301\240\003\002\001\000\241*0(\033\002nn\033\" uabigname02.rouen.francetelecom.fr\243\202\001\0320\202\001\026\240\003\002\001\022\241\003\002\001\a\242\202\001\b\004\202\001\004}\276$\220\240\2546\306\234\034\017W\354m\032\256\255\324\202\333\221\244\227\310bk\222\346\334\036\324\275a\262\360\221\315)\217\232Av\211\343\024\300\224\360\020ry\265\235\336\025\247\025\264\037B\3246\351\f\231\022\223h\254\277\241|dc\361\2037\305\2601\000\006\346\213qH\307\3452\r\306I\320\243\324\tu0b\367@\36\3727\224\214\002\\\317b2\217)\277\201X\356\244\370\310\246\004\304\362\262x\201\202\315\213\025{\361O\214*\216 \340\'\351@\331&\022\226\3727\240\233\201~\326\254\361\270*>z1\366G%98\0247\321\004\030?\242\203\3216\202%!z\336\002,\300\037\365\332\357\003\266\355RK\247,\026\271\274D\aC\335\222\304\313\203\'n\310z\350V\341\311\345\301\206\245/\224]\003\221\by]\325\305\333\314/P\277\035\032\000\273\266&\200n\225\352\375q\n\371\201\3704\366\000\344\'\352\352\244\201\3240\201\321\240\003\002\001\022\242\201\311\004\201\306\200\223\vd\273\252A\2259%\267\363\'\375\375u\000\346\307\377\346\276@\004I^\337\303d\232jv\347^[\034\276\355\260\260\355\263\016\027\035\033\034\327-\r\ "7\254\377\214\211\370\331jk:\360:\312@\262\221<Ce~\337u,\214\351\327I\024`Q*\235\351\245\314\211\004\340\2261\251\032\220\336\230/o\'>\310\255Iu@g\323Wp=\360yc\332\033\367\226\264\240l\205\2471O/\020Dm\371\377\273$W\342\273\034\027\307\341\016\226\3179\236\000\253m\361\3522[\211fZ\242\037A\334\v\240\033\242\337\203\245;_\003\020s\376\3509z\220\275\277\266\204\232,(=#hE\350\303\346\264\027q\002([+\373\201" auths { method: "KERBEROS" mechanism: "GSSAPI" protocol: "nn" serverId: "server2" } 15/12/21 10:47:56 DEBUG SaslRpcClient: Received SASL message state: CHALLENGE token: "`j\006\t*\206H\206\367\022\001\002\002\002\000o[0Y\240\003\002\001\005\241\003\002\001\017\242M0K\240\003\002\001\022\242D\004B\203\3422\031\250\024\265&\340\247\3161\254\022\300nTN\257\276\242<\211\361mw\353f_\344\374\017\306*B\265\005}\356\210\375\226\\\326\250\302\243\314\251\315]:\332Nx>\324\303;\317Y\332\317w" 15/12/21 10:47:56 DEBUG SaslRpcClient: Sending sasl message state: RESPONSE token: "" 15/12/21 10:47:56 DEBUG SaslRpcClient: Received SASL message state: CHALLENGE token: "\005\004\001\377\000\f\000\000\000\000\000\000\a2\025\223\001\001\000\000\025\254\334\363C\002\023\247h\272\270\240" 15/12/21 10:47:56 DEBUG SaslRpcClient: Sending sasl message state: RESPONSE token: "\005\004\000\377\000\f\000\000\000\000\000\000\016\\3\020\001\001\000\0005\322\312\330;\355\367U}N" 15/12/21 10:47:56 DEBUG SaslRpcClient: Received SASL message state: SUCCESS 15/12/21 10:47:56 DEBUG Client: Negotiated QOP is :auth 15/12/21 10:47:56 DEBUG Client: IPC Client (1212945869) connection to server2/192.168.200.24:8020 from pbeauvois@SANDBOX.HADOOP sending #59 15/12/21 10:47:56 DEBUG Client: IPC Client (1212945869) connection to server2/192.168.200.24:8020 from pbeauvois@SANDBOX.HADOOP: starting, having connections 2 15/12/21 10:47:56 DEBUG Client: IPC Client (1212945869) connection to server2/192.168.200.24:8020 from pbeauvois@SANDBOX.HADOOP got value #59 15/12/21 10:47:56 DEBUG ProtobufRpcEngine: Call: renewLease took 30ms 15/12/21 10:47:56 DEBUG LeaseRenewer: Lease renewed for client DFSClient_NONMAPREDUCE_-729245058_1 15/12/21 10:47:56 DEBUG LeaseRenewer: Lease renewer daemon for [DFSClient_NONMAPREDUCE_-729245058_1] with renew id 1 executed 15/12/21 10:47:56 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(ReviveOffers, false ) from Actor[akka: //sparkDriver/deadLetters] 15/12/21 10:47:56 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(ReviveOffers, false ) 15/12/21 10:47:56 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.682931 ms) AkkaMessage(ReviveOffers, false ) from Actor[akka: //sparkDriver/deadLetters] 15/12/21 10:48:01 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 15/12/21 10:48:01 WARN ClientCnxn: Session 0x0 for server null , unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) 15/12/21 10:48:01 DEBUG ClientCnxnSocketNIO: Ignoring exception during shutdown input java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:807) at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:426) at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:189) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) 15/12/21 10:48:01 DEBUG ClientCnxnSocketNIO: Ignoring exception during shutdown output java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:824) at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:434) at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:196) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) 15/12/21 10:48:01 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(ReviveOffers, false ) from Actor[akka: //sparkDriver/deadLetters] I tried to configure Spark differently. Setting SPARK_HOME and SPARK_CLASSPATH on spark-env.sh seem to be deprecated. So I configured spark-env.sh as the following: export HADOOP_HOME=/opt/application/Hadoop/current export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop export YARN_CONF_DIR=/opt/application/Hadoop/current/etc/hadoop #export SPARK_HOME=/opt/application/Spark/current #export SPARK_CLASSPATH=${SPARK_HOME}/lib_managed/jars/hbase-common.jar:${SPARK_HOME}/lib_managed/jars/hbase-server.jar:${SPARK_HOME}/lib_managed/jars/hbase-client.jar:${SPARK_HOME}/lib_managed/jars/hbase-protocol.jar:${SPARK_HOME}/lib_managed/jars/protobuf-java.jar:${SPARK_HOME}/lib_managed/jars/htrace-core-incubating.jar:${SPARK_HOME}/lib_managed/jars/hbase-annotations.jar:${SPARK_HOME}/lib_managed/jars/guava.jar:${SPARK_HOME}/lib_managed/jars/hadoop-common.jar #export HBASE_CONF_DIR=/opt/application/Hbase/current/conf and spark-defaults.conf: spark.driver.extraClassPath = /opt/application/Hbase/current/conf/ Then I launched spark-submit like that: [pbeauvois@uabigspark01 ~]$ spark-submit -v --master yarn-client --class org.apache.spark.examples.HBaseTest --jars /opt/application/Spark/current/lib_managed/jars/hbase-common.jar,/opt/application/Spark/current/lib_managed/jars/hbase-server.jar,/opt/application/Spark/current/lib_managed/jars/hbase-client.jar,/opt/application/Spark/current/lib_managed/jars/hbase-protocol.jar,/opt/application/Spark/current/lib_managed/jars/protobuf-java.jar,/opt/application/Spark/current/lib_managed/jars/htrace-core-incubating.jar,/opt/application/Spark/current/lib_managed/jars/hbase-annotations.jar,/opt/application/Spark/current/lib_managed/jars/guava.jar,/opt/application/Spark/current/lib_managed/jars/hadoop-common.jar /opt/application/Spark/current/examples/target/spark-examples_2.10-1.5.2.jar pbeauvois:noheader In that case, the error is the same as the one I posted in the description. As a conclusion, I can't use Spark to request HBase tables for the moment. Any ideas?
          Hide
          ybodnar Y Bodnar added a comment -

          Hi Pierre Beauvois

          It's weird there's no messages related to HBase tokens. According to yarn.Client "Attempting to fetch HBase security token." message should appear.

          Client.scala
          def obtainTokenForHBase(conf: Configuration, credentials: Credentials): Unit = {
              if (UserGroupInformation.isSecurityEnabled) {
                val mirror = universe.runtimeMirror(getClass.getClassLoader)
          
                try {
                  val confCreate = mirror.classLoader.
                    loadClass("org.apache.hadoop.hbase.HBaseConfiguration").
                    getMethod("create", classOf[Configuration])
                  val obtainToken = mirror.classLoader.
                    loadClass("org.apache.hadoop.hbase.security.token.TokenUtil").
                    getMethod("obtainToken", classOf[Configuration])
          
                  logDebug("Attempting to fetch HBase security token.")
          

          I would suggest to try 2 things:
          1. Check UserGroupInformation.isSecurityEnabled from your code. If it's false then no attempt to obtain ticket is being made
          2. Print HBaseConfiguration and check security related options (like hbase.scurity.authentication) to see if they're properly set and hbase-site.xml you provide is actually applied

          Show
          ybodnar Y Bodnar added a comment - Hi Pierre Beauvois It's weird there's no messages related to HBase tokens. According to yarn.Client "Attempting to fetch HBase security token." message should appear. Client.scala def obtainTokenForHBase(conf: Configuration, credentials: Credentials): Unit = { if (UserGroupInformation.isSecurityEnabled) { val mirror = universe.runtimeMirror(getClass.getClassLoader) try { val confCreate = mirror.classLoader. loadClass( "org.apache.hadoop.hbase.HBaseConfiguration" ). getMethod( "create" , classOf[Configuration]) val obtainToken = mirror.classLoader. loadClass( "org.apache.hadoop.hbase.security.token.TokenUtil" ). getMethod( "obtainToken" , classOf[Configuration]) logDebug( "Attempting to fetch HBase security token." ) I would suggest to try 2 things: 1. Check UserGroupInformation.isSecurityEnabled from your code. If it's false then no attempt to obtain ticket is being made 2. Print HBaseConfiguration and check security related options (like hbase.scurity.authentication) to see if they're properly set and hbase-site.xml you provide is actually applied
          Hide
          pbeauvois Pierre Beauvois added a comment -

          Hi Y Bodnar,
          I recently had the opportunity to do more tests on this feature. I installed Spark 1.6.0 on our cluster so I to I took advantage of this opportunity to reconfigure entirely Spark/HBase.

          Here is my new configuration:

          • configuration of the HBase security
            <property>
                <name>hbase.security.exec.permission.checks</name>
                <value>true</value>
            </property>
            
            <property>
                <name>hbase.security.visibility.mutations.checkauth</name>
                <value>false</value>
            </property>
            
            <property>
                <name>hbase.security.authentication</name>
                <value>kerberos</value>
            </property>
            
            <property>
                <name>hbase.security.authorization</name>
                <value>true</value>
            </property>
            
          • creation of a symbolic link to hbase-site.xml in the dir /opt/application/Spark/current/conf/
          • creation of symbolic links to HBase libraries in /opt/application/Spark/current/lib_managed/jars/
          • addition of these HBase libraries in the spark-defaults.conf
            # HBase path to the hbase main configuration (
            spark.driver.extraClassPath = /opt/application/Spark/current/lib_managed/jars/hive-hbase-handler.jar:/opt/application/Spark/current/lib_managed/jars/hbase-server.jar:/opt/application/Spark/current/lib_managed/jars/hbase-protocol.jar:/opt/application/Spark/current/lib_managed/jars/hbase-hadoop2-compat.jar:/opt/application/Spark/current/lib_managed/jars/hbase-client.jar:/opt/application/Spark/current/lib_managed/jars/hbase-common.jar:/opt/application/Spark/current/lib_managed/jars/hbase-annotations.jar:/opt/application/Spark/current/lib_managed/jars/guava.jar:/opt/application/Spark/current/lib_managed/jars/htrace-core-incubating.jar
            

          The good news is that I had been able to make it work but only in local mode.
          In local mode I did the following:

          $ spark-shell -v --master local
          ...
          scala> import org.apache.spark._
          import org.apache.spark._
          
          scala> import org.apache.spark.rdd.NewHadoopRDD
          import org.apache.spark.rdd.NewHadoopRDD
          
          scala> import org.apache.hadoop.fs.Path
          import org.apache.hadoop.fs.Path
          
          scala> import org.apache.hadoop.hbase.util.Bytes
          import org.apache.hadoop.hbase.util.Bytes
          
          scala> import org.apache.hadoop.hbase.HColumnDescriptor
          import org.apache.hadoop.hbase.HColumnDescriptor
          
          scala> import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor}
          import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor}
          
          scala> import org.apache.hadoop.hbase.client.{HBaseAdmin, Put, HTable, Result}
          import org.apache.hadoop.hbase.client.{HBaseAdmin, Put, HTable, Result}
          
          scala> import org.apache.hadoop.hbase.mapreduce.TableInputFormat
          import org.apache.hadoop.hbase.mapreduce.TableInputFormat
          
          scala> import org.apache.hadoop.hbase.io.ImmutableBytesWritable
          import org.apache.hadoop.hbase.io.ImmutableBytesWritable
          
          scala> val conf = HBaseConfiguration.create()
          conf: org.apache.hadoop.conf.Configuration = Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, hbase-default.xml, hbase-site.xml
          
          scala> conf.set("hbase.zookeeper.quorum", "myserver01.dns.fr:2181,myserver03.dns.fr:2181,myserver03.dns.fr:2181")
          
          scala> conf.set(TableInputFormat.INPUT_TABLE, "pbeauvois:death_causes")
          
          scala> val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat], classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable], classOf[org.apache.hadoop.hbase.client.Result])
          hBaseRDD: org.apache.spark.rdd.RDD[(org.apache.hadoop.hbase.io.ImmutableBytesWritable, org.apache.hadoop.hbase.client.Result)] = NewHadoopRDD[0] at newAPIHadoopRDD at <console>:40
          
          scala> hBaseRDD.collect().foreach(println)
          (33,keyvalues={1/case:count/1453278965864/Put/vlen=1/seqid=0, 1/case:death/1453278965830/Put/vlen=3/seqid=0, 1/case:location/1453278965901/Put/vlen=33/seqid=0, 1/case:year/1453278965670/Put/vlen=4/seqid=0, 1/case:zip/1453278965795/Put/vlen=5/seqid=0})
          (33,keyvalues={2/case:count/1453278966013/Put/vlen=1/seqid=0, 2/case:death/1453278965985/Put/vlen=3/seqid=0, 2/case:location/1453278966063/Put/vlen=33/seqid=0, 2/case:year/1453278965932/Put/vlen=4/seqid=0, 2/case:zip/1453278965958/Put/vlen=5/seqid=0})
          (33,keyvalues={3/case:count/1453278966173/Put/vlen=1/seqid=0, 3/case:death/1453278966148/Put/vlen=3/seqid=0, 3/case:location/1453278967718/Put/vlen=33/seqid=0, 3/case:year/1453278966101/Put/vlen=4/seqid=0, 3/case:zip/1453278966125/Put/vlen=5/seqid=0})
          

          I tried the same on "yarn-client" mode but I had the follwoing error stack:

          scala> hBaseRDD.collect().foreach(println)
          16/02/01 10:31:23 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
          org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, datanode04.dns.fr): java.lang.IllegalStateException: unread block data
                  at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2421)
                  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1382)
                  at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
                  at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
                  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
                  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
                  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
                  at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
                  at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
                  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:193)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)
          
          Driver stacktrace:
                  at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
                  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
                  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
                  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
                  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
                  at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
                  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
                  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
                  at scala.Option.foreach(Option.scala:236)
                  at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
                  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
                  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
                  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
                  at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
                  at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
                  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
                  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
                  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
                  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
                  at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:927)
                  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
                  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
                  at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
                  at org.apache.spark.rdd.RDD.collect(RDD.scala:926)
                  at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:43)
                  at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:48)
                  at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:50)
                  at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:52)
                  at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:54)
                  at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:56)
                  at $iwC$$iwC$$iwC$$iwC.<init>(<console>:58)
                  at $iwC$$iwC$$iwC.<init>(<console>:60)
                  at $iwC$$iwC.<init>(<console>:62)
                  at $iwC.<init>(<console>:64)
                  at <init>(<console>:66)
                  at .<init>(<console>:70)
                  at .<clinit>(<console>)
                  at .<init>(<console>:7)
                  at .<clinit>(<console>)
                  at $print(<console>)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:606)
                  at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
                  at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
                  at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
                  at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
                  at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
                  at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
                  at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
                  at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
                  at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
                  at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
                  at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
                  at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
                  at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
                  at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
                  at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
                  at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
                  at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
                  at org.apache.spark.repl.Main$.main(Main.scala:31)
                  at org.apache.spark.repl.Main.main(Main.scala)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:606)
                  at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
                  at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
                  at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
                  at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
                  at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
          Caused by: java.lang.IllegalStateException: unread block data
                  at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2421)
                  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1382)
                  at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
                  at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
                  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
                  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
                  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
                  at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
                  at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
                  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:193)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)
          

          Any ideas of the source of this error? Did I do a mistake in the configuration?

          Show
          pbeauvois Pierre Beauvois added a comment - Hi Y Bodnar , I recently had the opportunity to do more tests on this feature. I installed Spark 1.6.0 on our cluster so I to I took advantage of this opportunity to reconfigure entirely Spark/HBase. Here is my new configuration: configuration of the HBase security <property> <name>hbase.security.exec.permission.checks</name> <value> true </value> </property> <property> <name>hbase.security.visibility.mutations.checkauth</name> <value> false </value> </property> <property> <name>hbase.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.security.authorization</name> <value> true </value> </property> creation of a symbolic link to hbase-site.xml in the dir /opt/application/Spark/current/conf/ creation of symbolic links to HBase libraries in /opt/application/Spark/current/lib_managed/jars/ addition of these HBase libraries in the spark-defaults.conf # HBase path to the hbase main configuration ( spark.driver.extraClassPath = /opt/application/Spark/current/lib_managed/jars/hive-hbase-handler.jar:/opt/application/Spark/current/lib_managed/jars/hbase-server.jar:/opt/application/Spark/current/lib_managed/jars/hbase-protocol.jar:/opt/application/Spark/current/lib_managed/jars/hbase-hadoop2-compat.jar:/opt/application/Spark/current/lib_managed/jars/hbase-client.jar:/opt/application/Spark/current/lib_managed/jars/hbase-common.jar:/opt/application/Spark/current/lib_managed/jars/hbase-annotations.jar:/opt/application/Spark/current/lib_managed/jars/guava.jar:/opt/application/Spark/current/lib_managed/jars/htrace-core-incubating.jar The good news is that I had been able to make it work but only in local mode. In local mode I did the following: $ spark-shell -v --master local ... scala> import org.apache.spark._ import org.apache.spark._ scala> import org.apache.spark.rdd.NewHadoopRDD import org.apache.spark.rdd.NewHadoopRDD scala> import org.apache.hadoop.fs.Path import org.apache.hadoop.fs.Path scala> import org.apache.hadoop.hbase.util.Bytes import org.apache.hadoop.hbase.util.Bytes scala> import org.apache.hadoop.hbase.HColumnDescriptor import org.apache.hadoop.hbase.HColumnDescriptor scala> import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor} import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor} scala> import org.apache.hadoop.hbase.client.{HBaseAdmin, Put, HTable, Result} import org.apache.hadoop.hbase.client.{HBaseAdmin, Put, HTable, Result} scala> import org.apache.hadoop.hbase.mapreduce.TableInputFormat import org.apache.hadoop.hbase.mapreduce.TableInputFormat scala> import org.apache.hadoop.hbase.io.ImmutableBytesWritable import org.apache.hadoop.hbase.io.ImmutableBytesWritable scala> val conf = HBaseConfiguration.create() conf: org.apache.hadoop.conf.Configuration = Configuration: core- default .xml, core-site.xml, mapred- default .xml, mapred-site.xml, yarn- default .xml, yarn-site.xml, hdfs- default .xml, hdfs-site.xml, hbase- default .xml, hbase-site.xml scala> conf.set( "hbase.zookeeper.quorum" , "myserver01.dns.fr:2181,myserver03.dns.fr:2181,myserver03.dns.fr:2181" ) scala> conf.set(TableInputFormat.INPUT_TABLE, "pbeauvois:death_causes" ) scala> val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat], classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable], classOf[org.apache.hadoop.hbase.client.Result]) hBaseRDD: org.apache.spark.rdd.RDD[(org.apache.hadoop.hbase.io.ImmutableBytesWritable, org.apache.hadoop.hbase.client.Result)] = NewHadoopRDD[0] at newAPIHadoopRDD at <console>:40 scala> hBaseRDD.collect().foreach(println) (33,keyvalues={1/ case :count/1453278965864/Put/vlen=1/seqid=0, 1/ case :death/1453278965830/Put/vlen=3/seqid=0, 1/ case :location/1453278965901/Put/vlen=33/seqid=0, 1/ case :year/1453278965670/Put/vlen=4/seqid=0, 1/ case :zip/1453278965795/Put/vlen=5/seqid=0}) (33,keyvalues={2/ case :count/1453278966013/Put/vlen=1/seqid=0, 2/ case :death/1453278965985/Put/vlen=3/seqid=0, 2/ case :location/1453278966063/Put/vlen=33/seqid=0, 2/ case :year/1453278965932/Put/vlen=4/seqid=0, 2/ case :zip/1453278965958/Put/vlen=5/seqid=0}) (33,keyvalues={3/ case :count/1453278966173/Put/vlen=1/seqid=0, 3/ case :death/1453278966148/Put/vlen=3/seqid=0, 3/ case :location/1453278967718/Put/vlen=33/seqid=0, 3/ case :year/1453278966101/Put/vlen=4/seqid=0, 3/ case :zip/1453278966125/Put/vlen=5/seqid=0}) I tried the same on "yarn-client" mode but I had the follwoing error stack: scala> hBaseRDD.collect().foreach(println) 16/02/01 10:31:23 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, datanode04.dns.fr): java.lang.IllegalStateException: unread block data at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2421) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1382) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:193) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929) at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:927) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.RDD.collect(RDD.scala:926) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:43) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:48) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:50) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:52) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:54) at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:56) at $iwC$$iwC$$iwC$$iwC.<init>(<console>:58) at $iwC$$iwC$$iwC.<init>(<console>:60) at $iwC$$iwC.<init>(<console>:62) at $iwC.<init>(<console>:64) at <init>(<console>:66) at .<init>(<console>:70) at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>) at $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346) at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657) at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665) at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670) at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997) at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) at org.apache.spark.repl.Main$.main(Main.scala:31) at org.apache.spark.repl.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.IllegalStateException: unread block data at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2421) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1382) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:193) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Any ideas of the source of this error? Did I do a mistake in the configuration?
          Hide
          timeleft-- Younos Aboulnaga added a comment -

          I think this might be caused by a more general problem with the class path of the executor thread.

          Show
          timeleft-- Younos Aboulnaga added a comment - I think this might be caused by a more general problem with the class path of the executor thread.
          Hide
          ludochane Ludovic Chane added a comment -

          Hi Pierre Beauvois,
          I add the same problem and I fixed it by putting my hbase-site.xml in $SPARK_CONF_DIR (for me it is /etc/spark/conf).
          If you don't, Yarn Client does not have the good configuration (given your hbase-site.xml contains the good configuration ) and can't obtain HBase Kerberos Token for your Spark job.
          Passing hbase-site.xml with SPARK_CLASSPATH, --files or any other options did not work. Only putting it in $SPARK_CONF_DIR worked.

          Show
          ludochane Ludovic Chane added a comment - Hi Pierre Beauvois , I add the same problem and I fixed it by putting my hbase-site.xml in $SPARK_CONF_DIR (for me it is /etc/spark/conf). If you don't, Yarn Client does not have the good configuration (given your hbase-site.xml contains the good configuration ) and can't obtain HBase Kerberos Token for your Spark job. Passing hbase-site.xml with SPARK_CLASSPATH, --files or any other options did not work. Only putting it in $SPARK_CONF_DIR worked.
          Hide
          vanzin Marcelo Vanzin added a comment -

          I'm going to close this because I'm pretty sure this works (we have tests for it internally). The main things you have to look at are:

          • make sure Spark can see your hbase-site.xml file and any other needed config file (e.g. by placing it in $SPARK_CONF_DIR)
          • make sure HBase jars are in Spark's classpath.

          I believe in 2.2 HBase jars can even just be provided with --jars. Cluster mode might be a little trickier, but it also works (we use SPARK_DIST_CLASSPATH for that case).

          Show
          vanzin Marcelo Vanzin added a comment - I'm going to close this because I'm pretty sure this works (we have tests for it internally). The main things you have to look at are: make sure Spark can see your hbase-site.xml file and any other needed config file (e.g. by placing it in $SPARK_CONF_DIR ) make sure HBase jars are in Spark's classpath. I believe in 2.2 HBase jars can even just be provided with --jars . Cluster mode might be a little trickier, but it also works (we use SPARK_DIST_CLASSPATH for that case).

            People

            • Assignee:
              Unassigned
              Reporter:
              pbeauvois Pierre Beauvois
            • Votes:
              4 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development