[HIVE-7372] Select query gives unpredictable incorrect result when parallelism is greater than 1 [Spark Branch]

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: spark-branch
• Component/s: Spark
• Assignee: Chengxiang Li
• Reporter: Xuefu Zhang
• Created: 09/Jul/14 15:11
• Updated: 11/Jul/14 05:39
• Resolved: 11/Jul/14 05:38

Description

In SparkClient.java, if the following property is set, unpredictable, incorrect result may be observed.

    sparkConf.set("spark.default.parallelism", "1");

It's suspected that there are some concurrency issues, as Spark may process multiple datasets in a single JVM when parallelism is greater than 1 in order to use multiple cores.

NO PRECOMMIT TESTS. This is for spark branch only.

Issue Links

• is part of: HIVE-7292 Hive on Spark