Details
-
Bug
-
Status: Reopened
-
Major
-
Resolution: Unresolved
-
3.0.0, 3.0.1
-
None
Description
Following the PR that introduced the Prometheus sink, I downloaded the spark-3.0.1-bin-hadoop2.7.tgz (also tested with 3.0.0), uncompressed the tgz and created a file called metrics.properties adding this content:
*.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet
*.sink.prometheusServlet.path=/metrics/prometheus
master.sink.prometheusServlet.path=/metrics/master/prometheus
applications.sink.prometheusServlet.path=/metrics/applications/prometheus
Then I ran:
$ sbin/start-master.sh
$ sbin/start-slave.sh spark://`hostname`:7077
$ bin/spark-shell --master spark://`hostname`:7077 --files=./metrics.properties --conf spark.metrics.conf=./metrics.properties
The Spark shell opens without problems:
20/11/25 17:36:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://192.168.0.6:4040
Spark context available as 'sc' (master = spark://MacBook-Pro-de-Paulo-2.local:7077, app id = app-20201125173618-0002).
Spark session available as 'spark'.
Welcome to
____ __
/ _/ ____ ____/ /_
\ \/ _ \/ _ `/ __/ '/
/__/ ./,// //\ version 3.0.0
/_/
Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
And when I try to fetch prometheus metrics for driver, everything works fine:
$ curl -s http://localhost:4040/metrics/prometheus/ | head -n 5
metrics_app_20201125173618_0002_driver_BlockManager_disk_diskSpaceUsed_MB_Number{type="gauges"} 0
metrics_app_20201125173618_0002_driver_BlockManager_disk_diskSpaceUsed_MB_Value{type="gauges"} 0
metrics_app_20201125173618_0002_driver_BlockManager_memory_maxMem_MB_Number{type="gauges"} 732
metrics_app_20201125173618_0002_driver_BlockManager_memory_maxMem_MB_Value{type="gauges"} 732
metrics_app_20201125173618_0002_driver_BlockManager_memory_maxOffHeapMem_MB_Number{type="gauges"} 0
The problem appears when I try accessing master metrics, and I get the following problem:
$ curl -s http://localhost:8080/metrics/master/prometheus
<!DOCTYPE html><html>
<head>
<meta http-equiv="Content-type" content="text/html; charset=utf-8"/><link rel="stylesheet" href="/static/bootstrap.min.css" type="text/css"/><link rel="stylesheet" href="/static/vis-timeline-graph2d.min.css" type="text/css"/><link rel="stylesheet" href="/static/webui.css" type="text/css"/><link rel="stylesheet" href="/static/timeline-view.css" type="text/css"/><script src="/static/sorttable.js"></script><script src="/static/jquery-3.4.1.min.js"></script><script src="/static/vis-timeline-graph2d.min.js"></script><script src="/static/bootstrap-tooltip.js"></script><script src="/static/initialize-tooltips.js"></script><script src="/static/table.js"></script><script src="/static/timeline-view.js"></script><script src="/static/log-view.js"></script><script src="/static/webui.js"></script><script>setUIRoot('')</script>
<link rel="shortcut icon" href="/static/spark-logo-77x50px-hd.png"></link>
<title>Spark Master at spark://MacBook-Pro-de-Paulo-2.local:7077</title>
</head>
<body>
<div class="container-fluid">
<div class="row-fluid">
<div class="span12">
<h3 style="vertical-align: middle; display: inline-block;">
<a style="text-decoration: none" href="/">
<img src="/static/spark-logo-77x50px-hd.png"/>
<span class="version" style="margin-right: 15px;">3.0.0</span>
</a>
Spark Master at spark://MacBook-Pro-de-Paulo-2.local:7077
</h3>
</div>
</div>
<div class="row-fluid">
<div class="span12">
<ul class="unstyled">
<li><strong>URL:</strong> spark://MacBook-Pro-de-Paulo-2.local:7077</li>
...
Instead of the metrics I'm getting an HTML page. The same happens for all of those here:
$ curl -s http://localhost:8080/metrics/applications/prometheus/
$ curl -s http://localhost:8081/metrics/prometheus/
Instead, I expected metrics in prometheus metrics. All related JSON endpoints seem to be working fine.