[CARBONDATA-1489] Stabilize Array DataType Support

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.1.1
• Component/s: core, sql
• Assignee: Venkata Gollamudi
• Reporter: Venkata Gollamudi
• Created: 19/Sep/17 11:35
• Updated: 03/Feb/20 07:07

Description

Stabilize Array DataType Support. This is umbrella jira to track all related tasks.
Following cases needs to be handled:

Sub feature	Pending activity	Remarks
query, filter, NULL values, UDFs, Describe support	Test + Fix	Implementation in place needs to add test-cases and bug fix
Compaction support	Test + fix	As compaction works at byte level, no changes required. Needs to add test-cases
Insert into table	Develop	Source table data containing Array data needs to convert from spark datatype to string , as carbon takes string as input row
Support DDL for Array fields Dictionary include and Dictionary Exclude	Develop	Also needs to handle CarbonDictionaryDecoder to handle the same.
Support multilevel Array	Develop	currently DDL is validated to allow only 2 levels, remove this restriction
Support Array value to be a measure	Develop	Currently supports only dimensions
Support Alter table to add and remove Array column	Develop	implement DDL and requires default value handling
Projections of Array values push down to carbon	Develop	this is an optimization, when more number of values are present in Array
Filter Array values push down to carbon	Develop	this is an optimization, when more number of values are present in Array
Update Array values	Develop	update array value

Issue Links

• is related to: CARBONDATA-1457 Stabilize Struct DataType Support
• relates to: CARBONDATA-70 array of array complex dimension execution is failing
• relates to: CARBONDATA-2605 Complex DataType Enhancements
• links to: GitHub Pull Request #1441
• links to: GitHub Pull Request #1441

Sub-Tasks

• Load, query, filter, NULL values, UDFs, Describe support: Open
• Array type : Compaction support: Open
• Array type : insert into table support: Open
• Support DDL for Array fields Dictionary include and Dictionary Exclude: Open
• Support multilevel Array: Open
• Support Array type to be a measure: Open
• Support Alter table to add and remove Array column: Open
• Update Array values: Open