Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19814

RPC Server port is always random for spark

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0, 3.0.0, 2.4.0, 4.0.0
    • Fix Version/s: 4.0.0
    • Component/s: Spark
    • Labels:
      None

      Description

      RPC server port is always a random one. In fact, the problem is in RpcConfiguration.HIVE_SPARK_RSC_CONFIGS which doesn't include SPARK_RPC_SERVER_PORT.

       

      I've found this issue while trying to make hive-on-spark running inside docker.

       

      HIVE_SPARK_RSC_CONFIGS is called by HiveSparkClientFactory.initiateSparkConf > SparkSessionManagerImpl.setup and the latter call SparkClientFactory.initialize(conf) which initialize the rpc server. This RPCServer is then used to create the sparkClient which use the rpc server port as --remote-port arg. Since initiateSparkConf ignore SPARK_RPC_SERVER_PORT, then it will always be a random port.

        Attachments

        1. HIVE-19814.3.patch
          7 kB
          Bharathkrishna Guruvayoor Murali
        2. HIVE-19814.3.branch-3.patch
          8 kB
          Bharathkrishna Guruvayoor Murali
        3. HIVE-19814.3.branch-2.patch
          8 kB
          Bharathkrishna Guruvayoor Murali
        4. HIVE-19814.2.patch
          7 kB
          Bharathkrishna Guruvayoor Murali
        5. HIVE-19814.1.patch
          5 kB
          Bharathkrishna Guruvayoor Murali

          Activity

            People

            • Assignee:
              bharos92 Bharathkrishna Guruvayoor Murali
              Reporter:
              tiboun bounkong khamphousone
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: