Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12404

Ensure objects passed to StaticInvoke is Serializable

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.6.1, 2.0.0
    • Fix Version/s: 1.6.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Now `StaticInvoke` receives Any as a object and StaticInvoke can be serialized but sometimes the object passed is not serializable.

      For example, following code raises Exception because RowEncoder#extractorsFor invoked indirectly makes `StaticInvoke`.

      case class TimestampContainer(timestamp: java.sql.Timestamp)
      val rdd = sc.parallelize(1 to 2).map(_ => TimestampContainer(System.currentTimeMillis))
      val df = rdd.toDF
      val ds = df.as[TimestampContainer]
      val rdd2 = ds.rdd                                 <----------------- invokes extractorsFor indirectory
      

        Attachments

          Activity

            People

            • Assignee:
              sarutak Kousuke Saruta
              Reporter:
              sarutak Kousuke Saruta
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: