[SPARK-6868] Container link broken on Spark UI Executors page when YARN is set to HTTPS_ONLY

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.3.0
• Fix Version/s: 1.3.2, 1.4.0
• Component/s: YARN
• Assignee: Dean Chen
• Reporter: Dean Chen
• Created: 12/Apr/15 07:04
• Updated: 13/Apr/15 11:13
• Resolved: 13/Apr/15 11:13

Description

The stdout and stderr log links on the executor page will use the http:// prefix even if the node manager does not support http and only https via setting yarn.http.policy=HTTPS_ONLY.

Unfortunately the unencrypted http link in that case does not return a 404 but a binary file containing random binary chars. This causes a lot of confusion for the end user since it seems like the log file exists and is just filled with garbage. (see attached screenshot)

The fix is to prefix container log links with https:// instead of http:// if yarn.http.policy=HTTPS_ONLY. YARN's job page has this exact logic as seen here: https://github.com/apache/hadoop/blob/e1109fb65608a668cd53dc324dadc6f63a74eeb9/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobBlock.java#L108

Issue Links

• links to: [Github] Pull Request #5477 (deanchen)