[SPARK-17047] Spark 2 cannot create table when CLUSTERED.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.1.1, 2.2.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Flags: Important
• Assignee: Unassigned
• Reporter: Dr Mich Talebzadeh
• Created: 13/Aug/16 13:31
• Updated: 05/Oct/17 17:50
• Resolved: 05/Oct/17 17:50

Description

This does not work with CLUSTERED BY clause in Spark 2 now!

CREATE TABLE test.dummy2
(
ID INT
, CLUSTERED INT
, SCATTERED INT
, RANDOMISED INT
, RANDOM_STRING VARCHAR(50)
, SMALL_VC VARCHAR(10)
, PADDING VARCHAR(10)
)
CLUSTERED BY (ID) INTO 256 BUCKETS
STORED AS ORC
TBLPROPERTIES ( "orc.compress"="SNAPPY",
"orc.create.index"="true",
"orc.bloom.filter.columns"="ID",
"orc.bloom.filter.fpp"="0.05",
"orc.stripe.size"="268435456",
"orc.row.index.stride"="10000" )

scala> HiveContext.sql(sqltext)
org.apache.spark.sql.catalyst.parser.ParseException:
Operation not allowed: CREATE TABLE ... CLUSTERED BY(line 2, pos 0)

Issue Links

• blocks: SPARK-20901 Feature parity for ORC with Parquet
• is superceded by: SPARK-17729 Enable creating hive bucketed tables