Description
The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but the HoS still uses decides to use the RemoteHiveSparkClient rather than the LocalHiveSparkClient.
The issue is with the following check in HiveSparkClientFactory:
if (master.equals("local") || master.startsWith("local[")) { // With local spark context, all user sessions share the same spark context. return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf)); } else { return new RemoteHiveSparkClient(hiveconf, sparkConf); }
When master.startsWith("local[") it checks the value of spark.master and sees that it doesn't start with local[ and then decides to use the RemoteHiveSparkClient.
We should fix this so that the LocalHiveSparkClient is used. It should speed up some of the tests, and also makes qtests easier to debug since everything will now be run in the same process.
Attachments
Attachments
Issue Links
- links to