Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17081

Empty strings not preserved which causes SQLException: mismatching column value count

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • 1.4.1
    • 2.0.0
    • SQL
    • Important

    Description

      When writing a DataFrame that contains empty strings as values to an RDBMS, the query that is generated does not have the correct column count:

      CREATE TABLE demo(foo INTEGER, bar VARCHAR(10));
      -----
      case class Record(foo: Int, bar: String)
      val data = sc.parallelize(List(Record(1, ""))).toDF
      data.write.mode("append").jdbc(...)
      

      This causes:

      java.sql.SQLException: Column count doesn't match value count at row 1
      

      Proposal: leave empty strings as they are or convert these to NULL (although that may not be what's intended by the user, so make this configurable).

      Attachments

        Activity

          People

            Unassigned Unassigned
            hellstorm Ian Hellstrom
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: