Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      There is a config `spark.closure.serializer` that accepts exactly one value: the java serializer. This is because there are currently bugs in the Kryo serializer that make it not a viable candidate. This was uncovered by an unsuccessful attempt to make it work: SPARK-7708.

      My high level point is that the Java serializer has worked well for at least 6 Spark versions now, and it is an incredibly complicated task to get other serializers (not just Kryo) to work with Spark's closures. IMO the effort is not worth it and we should just remove this documentation and all the code associated with it.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                srowen Sean R. Owen
                Reporter:
                andrewor14 Andrew Or
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: