Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17081

Empty strings not preserved which causes SQLException: mismatching column value count

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • 1.4.1
    • 2.0.0
    • SQL
    • Important

    Description

      When writing a DataFrame that contains empty strings as values to an RDBMS, the query that is generated does not have the correct column count:

      CREATE TABLE demo(foo INTEGER, bar VARCHAR(10));
      -----
      case class Record(foo: Int, bar: String)
      val data = sc.parallelize(List(Record(1, ""))).toDF
      data.write.mode("append").jdbc(...)
      

      This causes:

      java.sql.SQLException: Column count doesn't match value count at row 1
      

      Proposal: leave empty strings as they are or convert these to NULL (although that may not be what's intended by the user, so make this configurable).

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            hellstorm Ian Hellstrom
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment