[CARBONDATA-562] Supporting Spark 1.6.3 integration in CarbonData

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.2.0-incubating
• Fix Version/s: 1.0.0-incubating
• Component/s: core
• Assignee: Naresh P R
• Reporter: Babulal
• Created: 25/Dec/16 05:51
• Updated: 22/Jan/17 03:44
• Resolved: 20/Jan/17 07:22

Description

Carbon is compiled with below command
mvn clean -P spark-1.6,provided -Dspark.version=1.6.3 -DskipTests package

start carbon context in yarn client mode

bin/spark-shell --master yarn-client --executor-memory 2G --executor-cores 2 --driver-memory 2G --num-executors 3

____ __
/ _/_ ___ ____/ /_
\ \/ _ \/ _ `/ __/ '/
/__/ ./_,// //_\ version 1.6.3
/_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111)
Type in expressions to have them evaluated.

scala> import org.apache.spark.sql.CarbonContext;
import org.apache.spark.sql.CarbonContext

scala> val cc=new CarbonContext(sc,"hdfs://hacluser/opt/CarbonStore");
e of dependencies)
scala.tools.reflect.ToolBoxError: reflective compilation has failed:

not enough arguments for constructor Optimizer: (conf: org.apache.spark.sql.catalyst.CatalystConf)org.apache.spark.sql.catalyst.optimizer.Optimizer.
Unspecified value parameter conf.
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.throwIfErrors(ToolBoxFactory.scala:314)
at org.apache.spark.sql.CodeGenerateFactory.<init>(CodeGenerateFactory.scala:32)
at org.apache.spark.sql.CodeGenerateFactory$.init(CodeGenerateFactory.scala:67)
at org.apache.spark.sql.CarbonContext.<init>(CarbonContext.scala:58)
at org.apache.spark.sql.CarbonContext.<init>(CarbonContext.scala:52)

Issue Links

• links to: GitHub Pull Request #558