[CARBONDATA-229] Array Index of bound exception thrown from dictionary look up while writing sort index file

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.2.0-incubating, 0.1.1-incubating
• Assignee: Manish Gupta
• Reporter: Manish Gupta
• Created: 08/Sep/16 08:14
• Updated: 24/Sep/16 10:56
• Resolved: 17/Sep/16 09:47

Description

Whenever we load dictionary data into memory, then in case of populating reverse dictionary object sometimes a chunk which has no value is also getting added to the dictionary chunk list. This is happening because the logic for dictionary chunk distribution in case of forward dictionary is not implemented for reverse dictionary and 0 size dictionary chunks are not getting removed while adding to the list of dictionary chunks.
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at org.apache.carbondata.core.cache.dictionary.DictionaryChunksWrapper.next(DictionaryChunksWrapper.java:92)
at org.apache.carbondata.core.writer.sortindex.CarbonDictionarySortInfoPreparator.prepareDictionarySortModels(CarbonDictionarySortInfoPreparator.java:120)
at org.apache.carbondata.core.writer.sortindex.CarbonDictionarySortInfoPreparator.getDictionarySortInfo(CarbonDictionarySortInfoPreparator.java:51)
at org.apache.carbondata.spark.tasks.SortIndexWriterTask.execute(SortIndexWriterTask.scala:44)
at org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD$$anon$1.<init>(CarbonGlobalDictionaryRDD.scala:387)
at org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD.compute(CarbonGlobalDictionaryRDD.scala:294)

Issue Links

• links to: GitHub Pull Request #143