[CARBONDATA-2420] Support string longer than 32000 characters

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.1
• Assignee: Unassigned
• Reporter: Chuanyin Xu
• Created: 01/May/18 02:26
• Updated: 01/Feb/19 05:49
• Resolved: 01/Feb/19 05:49

Description

Add a property in creating table 'long_string_columns' to support string columns that will contains more than 32000 characters.
Inside carbondata, it use an integer instead of short to store the length of bytes content.

Issue Links

• relates to: CARBONDATA-2764 Create table with duplicate long_string_column is successful, should be validated.
• relates to: CARBONDATA-2891 Job aborted while loading long string 32k data into carbon table from hive
• relates to: CARBONDATA-2920 For the Long string data, the local dictionary threshold is not reached even if the threshold condition is met
• relates to: CARBONDATA-2762 Long string column displayed as string in describe formatted
• relates to: CARBONDATA-2763 Create table with partition and no_inverted_index on long_string column is not blocked
• links to: GitHub Pull Request #2252
• links to: GitHub Pull Request #2379

Sub-Tasks

• long string columns should not be sort column: Resolved
• long string columns should not be dictionary columns: Resolved
• long_string_columns should be string columns: Resolved
• Support long_string_columns in sdk: Resolved
• Support long_string_columns property in dataframe writer: Resolved
• Support varchar datatype in DDL as longstring column: Open
• Support page size less than 32000 in CarbondataV3: Resolved
• Split to multiple pages if varchar column page exceeds 2GB/snappy limits: Resolved
• Complex datatype support long string: Open
• Support config long_string_columns when create datamap: Resolved
• create table with long_string_columns property: Resolved
• Fix loading problem using global/batch sort fails when table has long string columns: Resolved
• Fix data convertion problem for Varchar: Resolved
• show long_string_columns in desc table command: Resolved
• Block alter name/datatype of the long_string_columns: Closed
• Block alter table property of long_string_columns: Closed
• Add document for 32k feature: Resolved
• Fix data loading problem when table has complex column and long string column: Resolved
• Support longstring for streaming table: Open
• support long string columns with spark FileFormat and SDK with "long_string_columns" TableProperties: Resolved
• Fix bug in writing dataframe to carbon table while the field order is different: Resolved