[HIVE-18214] Flaky test: TestSparkClient

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Spark
• Target Version/s: 3.0.0
• Assignee: Sahil Takiar
• Reporter: Sahil Takiar
• Created: 04/Dec/17 22:09
• Updated: 22/May/18 23:14
• Resolved: 16/Jan/18 17:19

Description

Looks like there is a race condition in TestSparkClient#runTest. The test creates a RemoteDriver in memory, which creates a JavaSparkContext. A new JavaSparkContext is created for each test that is run. There is a race condition where the RemoteDriver isn't given enough time to shutdown, so when the next test starts running it creates another JavaSparkContext which causes an exception like org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243).

Issue Links

• links to: RB