[CARBONDATA-2148] Use Row parser to replace current default parser:CSVStreamParserImp

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.3.1
• Fix Version/s: 1.4.0, 1.3.1
• Component/s: data-load, spark-integration
• Assignee: Zhichao Zhang
• Reporter: Zhichao Zhang
• Created: 08/Feb/18 17:21
• Updated: 04/Mar/18 11:34
• Resolved: 04/Mar/18 11:34

Description

Currently the default value of 'carbon.stream.parser' is CSVStreamParserImp, it transforms InternalRow(0) to Array[Object], InternalRow(0) represents the value of one line which is received from Socket. When it receives data from Kafka, the schema of InternalRow is changed, either it need to assemble the fields of kafka data Row into a String and stored it as InternalRow(0), or define a new parser to convert kafka data Row to Array[Object]. It needs the same operation for every table.

Solution:
Use a new parser called RowStreamParserImpl as the default parser instead of CSVStreamParserImpl, this new parser will automatically convert InternalRow to Array[Object] according to the schema. In general, we will transform source data to a structed Row object, using this way, we do not need to define a parser for every table.

 

Issue Links

• links to: GitHub Pull Request #1959