[CARBONDATA-2199] Exception occurs when change the datatype of measure having sort_column

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0, 1.3.1
• Component/s: spark-integration
• Environment: spark 2.1
• Assignee: Jatin
• Reporter: Jatin
• Created: 23/Feb/18 10:48
• Updated: 28/Feb/18 04:19
• Resolved: 28/Feb/18 04:18

Description

Use a measure columns in sort_column and change the datatype of that columns

Steps to replicate

CREATE TABLE non_partitiontable7(id Int,vin String,phonenumber Long,area String,salary Int, country String,logdate date)STORED BY 'org.apache.carbondata.format'TBLPROPERTIES('SORT_COLUMNS'='id,vin','sort_scope'='global_sort');
insert into non_partitiontable7 select 1,'A42151477823',125371344,'OutSpace',10000,'China','2017-02-12';
insert into non_partitiontable7 select 1,'Y42151477823',125371344,'midasia',10000,'China','2017-02-13';
insert into non_partitiontable7 select 1,'B42151477823',125371346,'OutSpace',10000,'US','2018-02-12';
insert into non_partitiontable7 select 1,'C42151477823',125371348,'InnerSpace',10001,'UK','2019-02-12';
select * from non_partitiontable7;
alter table non_partitiontable7 add columns (c1 int);
select * from non_partitiontable7;
alter table non_partitiontable7 change id id bigint;
select * from non_partitiontable7;

Exception StackTrace

Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 16.0 failed 4 times, most recent failure: Lost task 1.3 in stage 16.0 (TID 80, BLR1000023654, executor 3): java.lang.IllegalArgumentException: Wrong length: 4, expected 8
at org.apache.carbondata.core.util.ByteUtil.explainWrongLengthOrOffset(ByteUtil.java:581)
at org.apache.carbondata.core.util.ByteUtil.toLong(ByteUtil.java:553)
at org.apache.carbondata.core.util.DataTypeUtil.getDataBasedOnRestructuredDataType(DataTypeUtil.java:847)
at org.apache.carbondata.core.datastore.chunk.store.impl.unsafe.UnsafeVariableLengthDimesionDataChunkStore.fillRow(UnsafeVariableLengthDimesionDataChunkStore.java:181)
at org.apache.carbondata.core.datastore.chunk.impl.VariableLengthDimensionDataChunk.fillConvertedChunkData(VariableLengthDimensionDataChunk.java:112)
at org.apache.carbondata.core.scan.result.AbstractScannedResult.fillColumnarNoDictionaryBatch(AbstractScannedResult.java:256)
at org.apache.carbondata.core.scan.collector.impl.DictionaryBasedVectorResultCollector.scanAndFillResult(DictionaryBasedVectorResultCollector.java:163)
at org.apache.carbondata.core.scan.collector.impl.RestructureBasedVectorResultCollector.collectVectorBatch(RestructureBasedVectorResultCollector.java:128)
at org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.processNextBatch(DataBlockIteratorImpl.java:65)
at org.apache.carbondata.core.scan.result.iterator.VectorDetailQueryResultIterator.processNextBatch(VectorDetailQueryResultIterator.java:46)
at org.apache.carbondata.spark.vectorreader.VectorizedCarbonRecordReader.nextBatch(VectorizedCarbonRecordReader.java:283)
at org.apache.carbondata.spark.vectorreader.VectorizedCarbonRecordReader.nextKeyValue(VectorizedCarbonRecordReader.java:171)
at org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.hasNext(CarbonScanRDD.scala:402)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.scan_nextBatch$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:395)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:234)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:228)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

 

Issue Links

• links to: GitHub Pull Request #1993