Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36009

Missing GraphX classes in registerKryoClasses util method

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.1.2
    • 3.3.0
    • GraphX
    • None

    Description

      VertexRDDImpl uses an RDD[ShippableVertexPartition[VD]] however, GraphXUtils.registerKryoClasses does not register ShippableVertexPartition. This means when running with spark.kryo.registrationRequired set to true, we get a "Class is not registered" exception. This is an issue as it prevents other unregistered classes from being discovered using spark.kryo.registrationRequired as the first unregistered class found halts the whole job. It also potentially decreases the serialised size of the RDD when using Kryo.

      Attachments

        Activity

          People

            matthewrj Matthew
            matthewrj Matthew
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: