Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6123

Parquet reader should use the schema of every file to create converter

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 1.5.0
    • SQL
    • None

    Description

      For two parquet files for the same table having an array column, if values of the array in one file was created when containsNull was true and those in another file was created when containsNull was false, the containsNull in the merged schema will be true and we cannot correctly read data from the table created with containsNull=false.

      Attachments

        Activity

          People

            lian cheng Cheng Lian
            yhuai Yin Huai
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: