Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15225

Connecting to HBase via newAPIHadoopRDD in PySpark gives org.apache.hadoop.hbase.client.RetriesExhaustedException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.98.4
    • None
    • mapreduce, spark
    • None

    • spark 1.6.0 , Hbase 0.98.4, kerberos, hbase.rpc.protection set to authentication.

    Description

      Unable to read HBase table into Spark with hbase security authentication set to kerberos. Seeing the following error.

      : org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=31, exceptions:
      Thu Feb 04 22:01:55 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:01:56 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:01:56 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:01:57 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:01:59 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:02:03 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:02:13 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:02:23 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:02:34 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:02:44 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:03:04 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:03:24 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:03:44 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:04:04 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:04:24 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:04:44 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      Thu Feb 04 22:05:04 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.IOException: Connection reset by peer
      .
      .
      .
      Thu Feb 04 22:09:46 CST 2016, org.apache.hadoop.hbase.client.RpcRetryingCaller@395327da, java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException

      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:129)
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)
      at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:282)
      at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:187)
      at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:182)
      at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:109)
      at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:738)
      at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:178)
      at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:82)
      at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:282)
      at org.apache.hadoop.hbase.client.HTable.getRegionLocations(HTable.java:616)
      at org.apache.hadoop.hbase.util.RegionSizeCalculator.<init>(RegionSizeCalculator.java:79)
      at org.apache.hadoop.hbase.util.RegionSizeCalculator.<init>(RegionSizeCalculator.java:64)
      at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:160)
      at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:115)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
      at scala.Option.getOrElse(Option.scala:120)
      at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
      at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
      at scala.Option.getOrElse(Option.scala:120)
      at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
      at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1277)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
      at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
      at org.apache.spark.rdd.RDD.take(RDD.scala:1272)
      at org.apache.spark.api.python.SerDeUtil$.pairRDDToPython(SerDeUtil.scala:202)
      at org.apache.spark.api.python.PythonRDD$.newAPIHadoopRDD(PythonRDD.scala:530)
      at org.apache.spark.api.python.PythonRDD.newAPIHadoopRDD(PythonRDD.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
      at py4j.Gateway.invoke(Gateway.java:259)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:207)
      at java.lang.Thread.run(Thread.java:744)
      Caused by: java.io.IOException: Call to d-767tfz1.target.com/10.66.241.13:60020 failed on local exception: java.io.EOFException
      at org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1484)
      at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1456)
      at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1656)
      at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1714)
      at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:29900)
      at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:308)
      at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:164)
      at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:59)
      at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
      ... 42 more
      Caused by: java.io.EOFException
      at java.io.DataInputStream.readInt(DataInputStream.java:392)
      at org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1071)
      at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:727)

      Attachments

        Activity

          People

            Unassigned Unassigned
            88.sanjay Sanjay Kumar
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: