Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.1.3, 3.2.0
-
None
Description
In many cases, users use Spark and Hive together. When a user creates a view via Spark, the table output columns will store in table properties, such as
After that, if the user runs the command "create or replace view" via Hive, to change the schema. The old table properties added by Spark are not cleaned by Hive. Then users read the table via Spark. The schema didn't change. It very confused users.
How to reproduce:
spark-sql>create table lajin_table (a int, b int) stored as parquet; spark-sql>create view lajin_view as select * from lajin_table; spark-sql> desc lajin_view; a int NULL NULL b int NULL NULL hive>desc lajin_view; a int b int hive>create or replace view lajin_view as select a, b, 3 as c from lajin_table; hive>desc lajin_view; a int b int c int spark-sql> desc lajin_view; -- not changed a int NULL NULL b int NULL NULL
Attachments
Attachments
Issue Links
- links to