[CARBONDATA-307] Support executor side scan using CarbonInputFormat

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.1.0-incubating
• Component/s: spark-integration
• Assignee: Unassigned
• Reporter: Jacky Li
• Created: 13/Oct/16 02:54
• Updated: 05/Apr/17 10:40

Description

Currently, there are two read path in carbon-spark module:
1. CarbonContext => CarbonDatasourceRelation => CarbonScanRDD => QueryExecutor
In this case, CarbonScanRDD uses CarbonInputFormat to get the split, and use QueryExecutor for scan.

2. SqlContext => CarbonDatasourceHadoopRelation => CarbonHadoopFSRDD => CarbonInputFormat(CarbonRecordReader) => QueryExecutor
In this case, CarbonHadoopFSRDD uses CarbonInputFormat to do both get split and scan

Because of this, there are unnecessary duplicate code, they need to be unified.
The target approach should be:
sqlContext/carbonContext => CarbonDatasourceHadoopRelation => CarbonScanRDD => CarbonInputFormat(CarbonRecordReader) => QueryExecutor

Sub-Tasks

• Use CarbonInputFormat in CarbonScanRDD compute: Resolved
• Support two types of ReadSupport in CarbonRecordReader: Open
• Unify two datasource: CarbonDatasourceHadoopRelation and CarbonDatasourceRelation: Open
• Update CarbonSource to use CarbonDatasourceHadoopRelation: Open
• Make CarbonContext to use standard Datasource strategy: Open