Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19983

Getting ValidationFailureSemanticException on 'INSERT OVEWRITE'

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Not A Problem
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      Hi, I am creating a DataFrame and registering that DataFrame as temp table using df.createOrReplaceTempView('mytable'). After that I try to write the content from 'mytable' into Hive table(It has partition) using the following query

      insert overwrite table
      myhivedb.myhivetable
      partition(testdate) // ( 1) : Note here : I have a partition named 'testdate'
      select
      Field1,
      Field2,
      ...
      TestDate //(2) : Note here : I have a field named 'TestDate' ; Both (1) & (2) have the same name
      from
      mytable

      when I execute this query, I am getting the following error

      Exception in thread "main" org.apache.hadoop.hive.ql.metadata.Table$ValidationFailureSemanticException: Partition spec

      {testdate=, TestDate=2013-01-01}

      Looks like I am getting this error because of the same field names ; ie testdate(the partition in Hive) & TestDate (The field in temp table 'mytable')

      Whereas if my partition name testdate is different from the fieldname(ie TestDate), the query executes successuflly. Example...

      insert overwrite table
      myhivedb.myhivetable
      partition(my_partition) //Note here the partition name is not 'testdate'
      select
      Field1,
      Field2,
      ...
      TestDate
      from
      mytable

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              myjirauid Rajkumar
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: