Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Invalid
-
2.0.0, 2.2.0, 2.3.0, 2.4.0, 3.0.0
-
None
-
None
Description
In Spark whenever user specifies location uri in create table without external keyword the table is treated as external table .
Because of this behavior following following problems has been observed
a) user will not able to set an external location for a managed table.
b) compatibility issue with hive/impala where the system allow managed table to specify location uri if user created table without 'EXTERNAL' keyword.
scala> spark.sql("""CREATE TABLE IF NOT EXISTS ext2 (name STRING) LOCATION 'D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/abc_orc13'"""); -chgrp: 'HTIPL-23270\None' does not match expected pattern for group Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH... res15: org.apache.spark.sql.DataFrame = [] scala> spark.sql("desc formatted ext2").show(false) +-----------------------------+---------------------------------------------------------------++------- |col_name|data_type|comment| +-----------------------------+---------------------------------------------------------------++------- |name|string|null| | | | | | # Detailed Table Information| | | |Database|default| | |Table|ext2| | |Owner|Administrator| | |Created Time|Wed May 01 21:52:57 IST 2019| | |Last Access|Thu Jan 01 05:30:00 IST 1970| | |Created By|Spark 2.4.1| | |Type|EXTERNAL| | |Provider|hive| | |Table Properties|[transient_lastDdlTime=1556727777]| | |Location|file:/D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/abc_orc13| | |Serde Library|org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe| | |InputFormat|org.apache.hadoop.mapred.TextInputFormat| | |OutputFormat|org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat| | |Storage Properties|[serialization.format=1]| | |Partition Provider|Catalog| |
Attachments
Issue Links
- links to