[CARBONDATA-1014] Refactor on data loading and encoding override

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Jacky Li
• Created: 03/May/17 08:38
• Updated: 23/May/18 12:00

Description

Refactor on current data loading flow to make it:
1. Use vectorized processing as early as possible
2. Make index build (sorting) CPU cache efficient, by using rowId and key column vector to sort
3. Open interface for format extension, including column encoding, compression, statistics.

Design doc will be posted in this JIRA soon.

Sub-Tasks

• Refactor write step to use ColumnarPage: Resolved
• Make sort step output ColumnPage: Open
• Add interface for column encoding and compression: Resolved
• Make ColumnPage use Unsafe: Resolved
• Add TablePage for data load process: Resolved
• change statistics to use exact type instead of Object: Resolved
• Use Snappy.rawCompression on unsafe data: Resolved
• Add SQL and dataframe option for encoding override: Open
• Change carbon data file definition for encoding override: Closed
• Add fixed length encoding for timestamp/date data type: Open
• Add encoding selection strategy for columns: Resolved