Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16726

Improve `Union/Intersect/Except` error messages on incompatible types

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.1.0
    • SQL
    • None

    Description

      Currently, `UNION/INTERSECT/EXCEPT` query on incompatible types shows a misleading error message like `unresolved operator Union`. We had better show a more correct message. This will help users in the situation of SPARK-16704

      Before

      scala> sql("select 1,2,3 union (select 1,array(2),3)")
      org.apache.spark.sql.AnalysisException: unresolved operator 'Union;
      scala> sql("select 1,2,3 intersect (select 1,array(2),3)")
      org.apache.spark.sql.AnalysisException: unresolved operator 'Intersect;
      scala> sql("select 1,2,3 except (select 1,array(2),3)")
      org.apache.spark.sql.AnalysisException: unresolved operator 'Except;
      

      After

      scala> sql("select 1,2,3 union (select 1,array(2),3)")
      org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. The first table has `[IntegerType, IntegerType, IntegerType]` and second table has `[IntegerType, ArrayType(IntegerType,false), IntegerType]`. The 2th column is incompatible;
      scala> sql("select 1,2,3 intersect (select 1,array(2),3)")
      org.apache.spark.sql.AnalysisException: Intersect can only be performed on tables with the compatible column types. The first table has `[IntegerType, IntegerType, IntegerType]` and second table has `[IntegerType, ArrayType(IntegerType,false), IntegerType]`. The 2th column is incompatible;
      scala> sql("select 1,2,3 except (select 1,array(2),3)")
      org.apache.spark.sql.AnalysisException: Except can only be performed on tables with the compatible column types. The first table has `[IntegerType, IntegerType, IntegerType]` and second table has `[IntegerType, ArrayType(IntegerType,false), IntegerType]`. The 2th column is incompatible;
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dongjoon Dongjoon Hyun
            dongjoon Dongjoon Hyun
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment