[HIVE-17270] Qtest results show wrong number of executors

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark
• Assignee: Peter Vary
• Reporter: Peter Vary
• Created: 08/Aug/17 16:13
• Updated: 22/May/18 23:59
• Resolved: 02/Nov/17 11:01

Description

The hive-site.xml shows, that the TestMiniSparkOnYarnCliDriver uses 2 cores, and 2 executor instances to run the queries. See: https://github.com/apache/hive/blob/master/data/conf/spark/yarn-client/hive-site.xml#L233

When reading the log files for the query tests, I see the following:

2017-08-08T07:41:03,315  INFO [0381325d-2c8c-46fb-ab51-423defaddd84 main] session.SparkSession: Spark cluster current has executors: 1, total cores: 2, memory per executor: 512M, memoryFraction: 0.4

See: http://104.198.109.242/logs/PreCommit-HIVE-Build-6299/succeeded/171-TestMiniSparkOnYarnCliDriver-insert_overwrite_directory2.q-scriptfile1.q-vector_outer_join0.q-and-17-more/logs/hive.log

When running the tests against a real cluster, I found that running an explain query for the first time I see 1 executor, but running it for the second time I see 2 executors.

Also setting some spark configuration on the cluster resets this behavior. For the first time I will see 1 executor, and for the second time I will see 2 executors again.

Sub-Tasks

• Set the number of executors based on config if client does not provide information: Resolved
• Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores: Closed