Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36200 Improve schema check logic before execution
  3. SPARK-36271

Hive SerDe and V1 Insert data to parquet/orc/avro need to check schema too

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.2.0
    • Component/s: SQL
    • Labels:
      None

      Description

      withTempPath { path =>
              spark.sql(
                s"""
                   |SELECT
                   |NAMED_STRUCT('ID', ID, 'IF(ID=1,ID,0)', IF(ID=1,ID,0), 'B', ABS(ID)) AS col1
                   |FROM v
                     """.stripMargin).write.mode(SaveMode.Overwrite).parquet(path.getCanonicalPath)
            }
      

        Attachments

        1. screenshot-1.png
          198 kB
          angerszhu

          Activity

            People

            • Assignee:
              angerszhuuu angerszhu
              Reporter:
              angerszhuuu angerszhu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: