Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3630

Identify cause of Kryo+Snappy PARSING_ERROR

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Done
    • 1.1.0, 1.2.0
    • None
    • Spark Core
    • None

    Description

      A recent GraphX commit caused non-deterministic exceptions in unit tests so it was reverted (see SPARK-3400).

      Separately, aash observed the same exception stacktrace in an application-specific Kryo registrator:

      com.esotericsoftware.kryo.KryoException: java.io.IOException: failed to uncompress the chunk: PARSING_ERROR(2)
      com.esotericsoftware.kryo.io.Input.fill(Input.java:142) com.esotericsoftware.kryo.io.Input.require(Input.java:169) 
      com.esotericsoftware.kryo.io.Input.readInt(Input.java:325) com.esotericsoftware.kryo.io.Input.readFloat(Input.java:624) 
      com.esotericsoftware.kryo.serializers.DefaultSerializers$FloatSerializer.read(DefaultSerializers.java:127) 
      com.esotericsoftware.kryo.serializers.DefaultSerializers$FloatSerializer.read(DefaultSerializers.java:117) 
      com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732) 
      com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) 
      com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) 
      com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
      ...
      

      This ticket is to identify the cause of the exception in the GraphX commit so the faulty commit can be fixed and merged back into master.

      Attachments

        Issue Links

          Activity

            People

              joshrosen Josh Rosen
              aash Andrew Ash
              Votes:
              9 Vote for this issue
              Watchers:
              34 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: