[CARBONDATA-2920] For the Long string data, the local dictionary threshold is not reached even if the threshold condition is met

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Labels: 32k Local_dictionary
• Environment: 3 node cluster
• Assignee: Unassigned
• Reporter: Prasanna Ravichandran
• Created: 07/Sep/18 15:30
• Updated: 07/Sep/18 15:33

Description

For the Long string data, the local dictionary threshold is not reached even if the threshold condition is met.

【Test step】：
1. Create table with long string column with local dictionary threshold as 1000.
2. Load more than 1000 distinct LONG data.
3. Check if the threshold is met.

Test queries:

drop table if exists 1klongdata;
create table 1klongdata(st string) stored by 'carbondata' TBLPROPERTIES('local_dictionary_enable'='true','local_dictionary_threshold'='1000','long_string_columns'='st');
load data inpath "hdfs://hacluster/user/prasanna/1005longdata.csv" into table 1klongdata options('fileheader'='st');

【Expected Output】：Once the local dictionary threshold is crossed, it should display as "Local Dictionary threshold reached for the column: col_name, Unable to generate dictionary value. Dictionary threshold reached" in executor log.
【Actual Output】：It is not printing the fallback details for long data even if the threshold limit is reached.

Issue Links

• is related to: CARBONDATA-2420 Support string longer than 32000 characters