Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11737

String may not be serialized correctly with Kyro

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.5.1, 1.6.0
    • Fix Version/s: 1.5.3, 1.6.0
    • Component/s: SQL
    • Labels:
      None

      Description

      When run in cluster mode, the driver may have different memory (and configs) than executor, also if Kyro is used, then string can not be collected back to driver:

      >>> sqlContext.range(10).selectExpr("repeat(cast(id as string), 9)").show()
      +----------------------------+
      |repeat(cast(id as string),9)|
      +----------------------------+
      |                   	0|
      |                   	1|
      |                   	2|
      |                   	3|
      |                   	4|
      |                   	5|
      |                   	6|
      |                   	7|
      |                   	8|
      |                   	9|
      +----------------------------+
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                davies Davies Liu
                Reporter:
                davies Davies Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: