[CARBONDATA-1798] class cast exception when loading data from a hive table to carbondata table

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Component/s: spark-integration
• Environment: spark2.1
• Assignee: anubhav tarar
• Reporter: anubhav tarar
• Created: 22/Nov/17 12:42
• Updated: 15/Jan/18 05:49
• Resolved: 15/Jan/18 05:49

Description

1. i created the hive tpch table named lineitem with around 10 gb data

CREATE TABLE LINEITEM ( L_ORDERKEY INT ,L_PARTKEY INT ,L_SUPPKEY INT ,L_LINENUMBER INT ,L_QUANTITY DECIMAL(15,2) ,L_EXTENDEDPRICE DECIMAL(15,2) ,L_DISCOUNT DECIMAL(15,2) ,L_TAX DECIMAL(15,2) ,L_RETURNFLAG STRING ,L_LINESTATUS STRING ,L_SHIPDATE Date ,L_COMMITDATE Date ,L_RECEIPTDATE Date ,L_SHIPINSTRUCT STRING ,L_SHIPMODE STRING ,L_COMMENT STRING) stored as orc;

2.tried to insert the data from that hive table into carbondata table

CREATE TABLE LINEITEM ( L_ORDERKEY INT ,L_PARTKEY INT ,L_SUPPKEY INT ,L_LINENUMBER INT ,L_QUANTITY DECIMAL(15,2) ,L_EXTENDEDPRICE DECIMAL(15,2) ,L_DISCOUNT DECIMAL(15,2) ,L_TAX DECIMAL(15,2) ,L_RETURNFLAG STRING ,L_LINESTATUS STRING ,L_SHIPDATE Date ,L_COMMITDATE Date ,L_RECEIPTDATE Date ,L_SHIPINSTRUCT STRING ,L_SHIPMODE STRING ,L_COMMENT STRING) stored by 'carbondata';

then i used INSERT INTO LINEITEM SELECT * FROM LINEITEM_ORC;

here are error logs

17/11/22 16:56:40 INFO BlockManagerInfo: Added broadcast_22_piece0 in memory on slave1:41401 (size: 25.2 KB, free: 3.0 GB)
17/11/22 16:59:12 WARN TaskSetManager: Lost task 0.0 in stage 15.0 (TID 20, hadoop-master, executor 2): org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException:
at org.apache.carbondata.processing.loading.sort.impl.UnsafeParallelReadMergeSorterImpl.sort(UnsafeParallelReadMergeSorterImpl.java:114)
at org.apache.carbondata.processing.loading.steps.SortProcessorStepImpl.execute(SortProcessorStepImpl.java:62)
at org.apache.carbondata.processing.loading.steps.DataWriterProcessorStepImpl.execute(DataWriterProcessorStepImpl.java:87)
at org.apache.carbondata.processing.loading.DataLoadExecutor.execute(DataLoadExecutor.java:50)
at org.apache.carbondata.spark.rdd.NewDataFrameLoaderRDD$$anon$2.<init>(NewCarbonDataLoadRDD.scala:391)
at org.apache.carbondata.spark.rdd.NewDataFrameLoaderRDD.internalCompute(NewCarbonDataLoadRDD.scala:354)
at org.apache.carbondata.spark.rdd.CarbonRDD.compute(CarbonRDD.scala:60)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.carbondata.processing.sort.exception.CarbonSortKeyAndGroupByException: org.apache.carbondata.processing.sort.exception.CarbonSortKeyAndGroupByException:
at org.apache.carbondata.processing.loading.sort.unsafe.merger.UnsafeIntermediateMerger.checkForFailure(UnsafeIntermediateMerger.java:188)
at org.apache.carbondata.processing.loading.sort.unsafe.merger.UnsafeIntermediateMerger.finish(UnsafeIntermediateMerger.java:171)
at org.apache.carbondata.processing.loading.sort.impl.UnsafeParallelReadMergeSorterImpl.sort(UnsafeParallelReadMergeSorterImpl.java:107)
... 14 more
Caused by: java.util.concurrent.ExecutionException: org.apache.carbondata.processing.sort.exception.CarbonSortKeyAndGroupByException:
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.carbondata.processing.loading.sort.unsafe.merger.UnsafeIntermediateMerger.checkForFailure(UnsafeIntermediateMerger.java:185)
... 16 more
Caused by: org.apache.carbondata.processing.sort.exception.CarbonSortKeyAndGroupByException:
at org.apache.carbondata.processing.loading.sort.unsafe.merger.UnsafeIntermediateFileMerger.call(UnsafeIntermediateFileMerger.java:142)
at org.apache.carbondata.processing.loading.sort.unsafe.merger.UnsafeIntermediateFileMerger.call(UnsafeIntermediateFileMerger.java:45)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to [B
at org.apache.carbondata.processing.loading.sort.unsafe.merger.UnsafeIntermediateFileMerger.writeDataTofile(UnsafeIntermediateFileMerger.java:333)
at org.apache.carbondata.processing.loading.sort.unsafe.merger.UnsafeIntermediateFileMerger.call(UnsafeIntermediateFileMerger.java:113)
... 5 more

17/11/22 16:59:12 INFO TaskSetManager: Starting task 0.1 in stage 15.0 (TID 22, hadoop-master, executor 2, partition 0, NODE_LOCAL, 7696 bytes)