[CARBONDATA-1175] Streaming Ingest: Write path data conversion/transformation

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.3.0
• Component/s: core, data-load, data-query, examples, file-format, spark-integration, sql
• Assignee: Unassigned
• Reporter: Aniket Adnaik
• Created: 14/Jun/17 19:11
• Updated: 03/Feb/18 09:47
• Resolved: 03/Feb/18 09:47

Description

Streaming Ingest: Write path Data conversion/Transformation

• input data is a byte stream in catalyst InternalRow format (row major), which needs to be converted to column format
• column converter and corresponding iterators needs to be created before invoking carbon layer load path
• various carbon properties needs to be set, for example SORT_SCOPE (to skip sorting), blocket size, skip global dictionary creation etc