[SPARK-9761] Inconsistent metadata handling with ALTER TABLE

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.4.1
• Component/s: SQL
• Labels: bulk-closed hive sql

Environment

Ubuntu on AWS

• Assignee: Unassigned
• Reporter: Simeon Simeonov
• Created: 09/Aug/15 03:09
• Updated: 21/May/19 04:33
• Resolved: 21/May/19 04:33

Description

Schema changes made with ALTER TABLE are not shown in DESCRIBE TABLE. The table in question was created with HiveContext.read.json().

Steps:

1. alter table dimension_components add columns (z string); succeeds.
2. describe dimension_components; does not show the new column, even after restarting spark-sql.
3. A second alter table dimension_components add columns (z string); fails with RROR exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: Duplicate column name: z

Full spark-sql output here.

Issue Links

• is related to: SPARK-9764 Spark SQL uses table metadata inconsistently