[HIVE-19814] RPC Server port is always random for spark

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0, 2.4.0, 3.0.0, 4.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Spark
• Assignee: Bharath Krishna
• Reporter: bounkong khamphousone
• Created: 06/Jun/18 08:01
• Updated: 17/Nov/22 08:55
• Resolved: 19/Nov/18 18:34

Description

RPC server port is always a random one. In fact, the problem is in RpcConfiguration.HIVE_SPARK_RSC_CONFIGS which doesn't include SPARK_RPC_SERVER_PORT.

 

I've found this issue while trying to make hive-on-spark running inside docker.

 

HIVE_SPARK_RSC_CONFIGS is called by HiveSparkClientFactory.initiateSparkConf > SparkSessionManagerImpl.setup and the latter call SparkClientFactory.initialize(conf) which initialize the rpc server. This RPCServer is then used to create the sparkClient which use the rpc server port as --remote-port arg. Since initiateSparkConf ignore SPARK_RPC_SERVER_PORT, then it will always be a random port.