[CARBONDATA-3422] fix missing complex dimensions when prepare the data from raw object

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.3
• Fix Version/s: 1.6.0
• Component/s: data-load, spark-integration
• Assignee: Unassigned
• Reporter: lamber-ken
• Created: 10/Jun/19 06:57
• Updated: 30/Apr/20 11:40
• Resolved: 30/Apr/20 11:40

Description

When use compact command for streaming table which contains complex column like array, it'll thow ArrayIndexOutOfBoundsException.

Compact command

 

ALTER TABLE streaming1.stream_table_filter_complex COMPACT 'close_streaming'

Stackstrace

java.lang.ArrayIndexOutOfBoundsException: 9
    at org.apache.carbondata.processing.loading.sort.SortStepRowHandler.packNoSortFieldsToBytes(SortStepRowHandler.java:808)
    at org.apache.carbondata.processing.loading.sort.SortStepRowHandler.writeRawRowAsIntermediateSortTempRowToOutputStream(SortStepRowHandler.java:481)
    at org.apache.carbondata.processing.sort.sortdata.SortDataRows.writeDataToFile(SortDataRows.java:244)
    at org.apache.carbondata.processing.sort.sortdata.SortDataRows.startSorting(SortDataRows.java:222)
    at org.apache.carbondata.processing.merger.CompactionResultSortProcessor.processResult(CompactionResultSortProcessor.java:257)
    at org.apache.carbondata.processing.merger.CompactionResultSortProcessor.execute(CompactionResultSortProcessor.java:175)
    at org.apache.carbondata.spark.rdd.StreamHandoffRDD.internalCompute(StreamHandoffRDD.scala:123)
    at org.apache.carbondata.spark.rdd.CarbonRDD.compute(CarbonRDD.scala:82)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)

 

 

 

 

Issue Links

• links to: GitHub Pull Request #3270