Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19814

RPC Server port is always random for spark

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0, 2.4.0, 3.0.0, 4.0.0
    • 4.0.0, 4.0.0-alpha-1
    • Spark
    • None

    Description

      RPC server port is always a random one. In fact, the problem is in RpcConfiguration.HIVE_SPARK_RSC_CONFIGS which doesn't include SPARK_RPC_SERVER_PORT.

       

      I've found this issue while trying to make hive-on-spark running inside docker.

       

      HIVE_SPARK_RSC_CONFIGS is called by HiveSparkClientFactory.initiateSparkConf > SparkSessionManagerImpl.setup and the latter call SparkClientFactory.initialize(conf) which initialize the rpc server. This RPCServer is then used to create the sparkClient which use the rpc server port as --remote-port arg. Since initiateSparkConf ignore SPARK_RPC_SERVER_PORT, then it will always be a random port.

      Attachments

        1. HIVE-19814.3.patch
          7 kB
          Bharath Krishna
        2. HIVE-19814.3.branch-3.patch
          8 kB
          Bharath Krishna
        3. HIVE-19814.3.branch-2.patch
          8 kB
          Bharath Krishna
        4. HIVE-19814.2.patch
          7 kB
          Bharath Krishna
        5. HIVE-19814.1.patch
          5 kB
          Bharath Krishna

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            bharos92 Bharath Krishna Assign to me
            tiboun bounkong khamphousone
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment