Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19641

JSON schema inference in DROPMALFORMED mode produces incorrect schema

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.0
    • 2.2.0
    • SQL
    • None

    Description

      In DROPMALFORMED mode the inferred schema may incorrectly contain no columns. This occurs when one document contains a valid JSON value (such as a string or number) and the other documents contain objects or arrays.

      When the default case in JsonInferSchema.compatibleRootType is reached when merging a StringType and a StructType the resulting type will be a StringType, which is then discarded because a StructType is expected.

      Attachments

        Activity

          People

            gurwls223 Hyukjin Kwon
            NathanHowell Nathan Howell
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: