[SPARK-28436] Throw better exception when datasource's schema is not equal to user-specific shema

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.3.0, 2.4.3
• Component/s: SQL
• Assignee: Unassigned
• Reporter: ShuMing Li
• Created: 18/Jul/19 09:34
• Updated: 25/Aug/19 05:39
• Resolved: 25/Aug/19 05:39

Description

When this exception is thrown, users cannot find what's the difference between datasource's original schema and user-specific schema, and maybe very confused when meet the exception below.

org.apache.spark.sql.AnalysisException: org.apache.spark.odps.datasource does not allow user-specified schemas.
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:347)
    at org.apache.spark.sql.execution.command.CreateDataSourceTableCommand.run(createDataSourceTables.scala:78)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
    at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
    at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
    at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3270)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:83)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3269)
    at org.apache.spark.sql.Dataset.<init>(Dataset.scala:190)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:75)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:653)
    at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:714)

Issue Links

• is duplicated by: SPARK-28438 Ignore metadata's(comments) difference when comparing datasource's schema and user-specific schema
• links to: GitHub Pull Request #25192