Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6868

Container link broken on Spark UI Executors page when YARN is set to HTTPS_ONLY

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.3.0
    • Fix Version/s: 1.3.2, 1.4.0
    • Component/s: YARN
    • Labels:
      None

      Description

      The stdout and stderr log links on the executor page will use the http:// prefix even if the node manager does not support http and only https via setting yarn.http.policy=HTTPS_ONLY.

      Unfortunately the unencrypted http link in that case does not return a 404 but a binary file containing random binary chars. This causes a lot of confusion for the end user since it seems like the log file exists and is just filled with garbage. (see attached screenshot)

      The fix is to prefix container log links with https:// instead of http:// if yarn.http.policy=HTTPS_ONLY. YARN's job page has this exact logic as seen here: https://github.com/apache/hadoop/blob/e1109fb65608a668cd53dc324dadc6f63a74eeb9/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobBlock.java#L108

        Attachments

          Activity

            People

            • Assignee:
              deanchen Dean Chen
              Reporter:
              deanchen Dean Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: