[SPARK-26234] Column list specification in INSERT statement

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.3.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Joby Joje
• Created: 30/Nov/18 13:13
• Updated: 30/Nov/18 14:48
• Resolved: 30/Nov/18 14:21

Description

While trying to OVERWRITE the Hive table with specific columns from Spark(Pyspark) using a dataframe getting the below error

pyspark.sql.utils.ParseException: u"\nmismatched input 'col1' expecting
Unknown macro: {'(', 'SELECT', 'FROM', 'VALUES', 'TABLE', 'INSERT', 'MAP', 'REDUCE'}
(line 1, pos 36)\n\n== SQL ==\ninsert into table DB.TableName (Col1, Col2, Col3) select Col1, Col2, Col3 FROM dataframe\n------------------------------------^^^\n"

sparkSession.sql("insert into table DB.TableName (Col1, Col2, Col3) select Col1, Col2, Col3 FROM dataframe")

But on trying the same via Hive Terminal goes through fine.

Please check the below link to get more info on the same.

https://stackoverflow.com/questions/53517671/column-list-specification-in-insert-overwrite-statement

 

Issue Links

• duplicates: SPARK-20845 Support specification of column names in INSERT INTO