[PIG-4970] Remove the deserialize and serialization of JobConf in code for spark mode

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: liyunzhang
• Reporter: liyunzhang
• Created: 12/Aug/16 05:37
• Updated: 21/Jun/17 09:18
• Resolved: 24/Aug/16 16:04

Description

Now we use KryoSerializer to serialize the jobConf in SparkLauncher. then
deserialize it in ForEachConverter, StreamConverter. We deserialize and serialize the jobConf in order to make jobConf available in spark executor thread.

We can refactor it in following ways:
1. Let spark to broadcast the jobConf in sparkContext.newAPIHadoopRDD. Here not create a new jobConf and load properties from PigContext but directly use jobConf from SparkLauncher.
2. get jobConf in org.apache.pig.backend.hadoop.executionengine.spark.running.PigInputFormatSpark#createRecordReader

Issue Links

• links to: review board