[SPARK-24302] when using spark persist(),"KryoException:IndexOutOfBoundsException" happens

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 1.6.0
• Component/s: Input/Output
• Labels: apache-spark
• Assignee: Unassigned
• Reporter: yijukang
• Created: 17/May/18 02:25
• Updated: 12/Dec/22 18:11
• Resolved: 21/May/18 03:05

Description

my operation is using spark to insert RDD data into Hbase like this:

------------------------------

localData.persist()
localData.saveAsNewAPIHadoopDataset(jobConf.getConfiguration)

--------------------------------------

this way throw Exception:

   com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException:index:99, Size:6

Serialization trace:

    familyMap (org.apache.hadoop.hbase.client.Put)

  at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)

   at com.esotericsoftware.kryo.kryo.readClassAndObject(Kryo.java:729)

   at com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:42)

   at com.twitter.chill.Tuple2Serializer.read(TupleSerializers.scala:33)

   at com.esotericsoftware.kryo.kryo.readClassAndObject(Kryo.java:729)

 

when i deal with this:

-----------------------------
localData.saveAsNewAPIHadoopDataset(jobConf.getConfiguration)

--------------------------------------

it works well，what the persist() method did?