[CARBONDATA-322] Integration with spark 2.x

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.2.0-incubating
• Fix Version/s: 1.0.0-incubating
• Component/s: spark-integration
• Assignee: Fei Wang
• Reporter: Fei Wang
• Created: 18/Oct/16 07:38
• Updated: 20/Jan/17 04:05
• Resolved: 20/Jan/17 04:05

Description

Since spark 2.0 released. there are many nice features such as more efficient parser, vectorized execution, adaptive execution.
It is good to integrate with spark 2.x

current integration up to Spark v1.6 is tightly coupled with spark, we would like to cleanup the interface with following design points in mind:

1. decoupled with Spark, integration based on Spark's v2 datasource API
2. Enable vectorized carbon reader
3. Support saving DataFrame to Carbondata file through Carbondata's output format.
...

Sub-Tasks

• Clean partitioner in RDD package: Resolved
• Clean up carbonTableSchema.scala before moving to spark-common package: Resolved
• Extract spark-common module: Resolved
• spark 2 stable datasource api integration: Resolved
• spark2 decimal issue: Resolved
• do not use runnablecommand in spark2: Resolved
• Use carbon property to get the store path/kettle home: Resolved
• Executor can not get the read support class: Resolved
• Depends on more stable class of spark in spark2: Resolved