Details
-
Bug
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
1.6.0
-
None
-
None
-
Spark 2.1
Description
【Test step】：
From Spark-beeline user creates a table.
For the table user creates a datamap.
User loads data.
User alters the table and sets sort columns.
User loads data.
0: jdbc:hive2://10.18.98.225:22550/default> CREATE TABLE uniqdata(CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','sort_columns'='CUST_NAME');
---------+
| Result |
---------+
---------+
No rows selected (1.663 seconds)
0: jdbc:hive2://10.18.98.225:22550/default> create datamap uniqdata_agg on table uniqdata using 'preaggregate' as select cust_name, avg(cust_id) from uniqdata group by cust_id, cust_name;
---------+
| Result |
---------+
---------+
No rows selected (4.36 seconds)
0: jdbc:hive2://10.18.98.225:22550/default> LOAD DATA INPATH 'hdfs://hacluster/chetan/2000_UniqData.csv' into table uniqdata OPTIONS('DELIMITER'=',', 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
---------+
| Result |
---------+
---------+
No rows selected (17.388 seconds)
0: jdbc:hive2://10.18.98.225:22550/default> alter table uniqdata set tblproperties('sort_columns'='ACTIVE_EMUI_VERSION','sort_scope'='global_sort');
---------+
| Result |
---------+
---------+
No rows selected (3.95 seconds)
0: jdbc:hive2://10.18.98.225:22550/default> LOAD DATA INPATH 'hdfs://hacluster/chetan/2000_UniqData.csv' into table uniqdata OPTIONS('DELIMITER'=',', 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
---------+
| Result |
---------+
---------+
No rows selected (15.33 seconds)
User executes major compaction.
0: jdbc:hive2://10.18.98.225:22550/default> alter table uniqdata compact 'major';
【Expected Output】：Major compaction should be success when data loaded for table with preaggregate datamap after sort columns modification
【Actual Output】：Major compaction fails with error.
0: jdbc:hive2://10.18.98.225:22550/default> alter table uniqdata compact 'major';
Error: org.apache.spark.sql.AnalysisException: Compaction failed. Please check logs for more info. Exception in compaction Compaction failed. Please check logs for more info. Exception in compaction Query On DataMap not supported because carbon.query.directQueryOnDataMap.enabled is false. Please change the value to true by set command or other if you want to query on DataMap.;;; (state=,code=0)