[CARBONDATA-2846] compaction is failing for table created in S3/OBS

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: data-query
• Assignee: Unassigned
• Reporter: Rahul Singha
• Created: 09/Aug/18 11:46
• Updated: 09/Aug/18 11:46

Description

Steps:

1.create table brinjal (imei string,AMSize string,channelsId string,ActiveCountry string, Activecity string,gamePointId double,deviceInformationId double,productionDate Timestamp,deliveryDate timestamp,deliverycharge double) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES('table_blocksize'='1');

2.LOAD DATA INPATH 'hdfs://hacluster/user/rahul/vardhandaterestruct.csv' INTO TABLE brinjal OPTIONS('DELIMITER'=',', 'QUOTECHAR'= '"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'= 'imei,deviceInformationId,AMSize,channelsId,ActiveCountry,Activecity,gamePointId,productionDate,deliveryDate,deliverycharge');

3.LOAD DATA INPATH 'hdfs://hacluster/user/rahul/vardhandaterestruct.csv' INTO TABLE brinjal OPTIONS('DELIMITER'=',', 'QUOTECHAR'= '"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'= 'imei,deviceInformationId,AMSize,channelsId,ActiveCountry,Activecity,gamePointId,productionDate,deliveryDate,deliverycharge');

4.alter table brinjal compact 'major';

Expected Result:Compaction should be successful

Actual Result:

alter table brinjal compact 'minor';
Error: org.apache.spark.sql.AnalysisException: Compaction failed. Please check logs for more info. Exception in compaction Job aborted due to stage failure: Task 0 in stage 100.0 failed 4 times, most recent failure: Lost task 0.3 in stage 100.0 (TID 349, BLR1000025336, executor 116): org.apache.spark.sql.AnalysisException: Exception occurred in query execution :: org.apache.carbondata.core.scan.result.iterator.RawResultIterator.<init>(Lorg/apache/carbondata/common/CarbonIterator;Lorg/apache/carbondata/core/datastore/block/SegmentProperties;Lorg/apache/carbondata/core/datastore/block/SegmentProperties;Z)V;
at org.apache.spark.sql.util.CarbonException$.analysisException(CarbonException.scala:23)
at org.apache.carbondata.spark.rdd.CarbonMergerRDD$$anon$1.<init>(CarbonMergerRDD.scala:190)
at org.apache.carbondata.spark.rdd.CarbonMergerRDD.internalCompute(CarbonMergerRDD.scala:85)
at org.apache.carbondata.spark.rdd.CarbonRDD.compute(CarbonRDD.scala:78)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:; (state=,code=0)