Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36200 Improve schema check logic before execution
  3. SPARK-33865

When HiveDDL, we need check avro schema too like parquet & orc

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • SQL
    • None

    Description

      Caused by: org.apache.avro.SchemaParseException: Illegal initial character: (IF((1 = 1), 1, 0))
      	at org.apache.avro.Schema.validateName(Schema.java:1147)
      	at org.apache.avro.Schema.access$200(Schema.java:81)
      	at org.apache.avro.Schema$Field.<init>(Schema.java:403)
      	at org.apache.avro.Schema$Field.<init>(Schema.java:396)
      	at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.createAvroField(TypeInfoToSchema.java:76)
      	at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.convert(TypeInfoToSchema.java:61)
      	at org.apache.hadoop.hive.serde2.avro.AvroSerDe.getSchemaFromCols(AvroSerDe.java:170)
      	at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:114)
      	at org.apache.spark.sql.hive.execution.HiveOutputWriter.<init>(HiveFileFormat.scala:121)
      	at org.apache.spark.sql.hive.execution.HiveFileFormat$$anon$1.newInstance(HiveFileFormat.scala:104)
      	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:126)
      
      

      Attachments

        Activity

          People

            angerszhuuu angerszhu
            angerszhuuu angerszhu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: