Description
Unlike MapReduce JobHistory Server, Spark history server isn't rewriting container log URL's to point to the aggregated yarn.log.server.url location and relies on the NodeManager webUI to trigger a redirect. This fails when the NM is down. Note that NM may be down permanently after decommissioning in traditional environments or when used in a cloud environment such as AWS EMR where either worker nodes are taken away with autoscale, the whole cluster is used to run a single job.
Attachments
Issue Links
- is duplicated by
-
SPARK-26311 [YARN] New feature: custom log URL for stdout/stderr
- Resolved
- is related to
-
YARN-7847 Provide permalinks for container logs
- Open
- links to