Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22914

Subbing for spark.history.ui.port does not resolve by default

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.1
    • Fix Version/s: 2.3.0
    • Component/s: Deploy
    • Labels:
      None

      Description

      In order not to hardcode SHS web ui port and not duplicate information that is already configured we might be inclined to define spark.yarn.historyServer.address as

      http://${hadoopconf-yarn.resourcemanager.hostname}:${spark.history.ui.port}

      However, since spark.history.ui.port is not registered its resolution fails when it's not explicitly set in the deployed spark conf.

        Attachments

          Activity

            People

            • Assignee:
              jira.shegalov Gera Shegalov
              Reporter:
              jira.shegalov Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: