Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
2.3.0
-
None
-
None
-
SW
hadoop-3.0.0
Spark 2.3.0 (git revision a0d7949896) built for Hadoop 2.7.3Nodes
jupyter
proxyserver
historyserver
resourcemanager
nodemanager (http://9d581cfb391f:8042)Config of sparkcontext
spark.app.id=application_1519904229414_0011
spark.app.name=Spark shell
spark.driver.appUIAddress=http://jupyter:4040
spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/tmp/driver_log4j.properties
spark.driver.host=jupyter
spark.driver.port=35813
spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/tmp/driver_log4j.properties
spark.executor.id=driver
spark.home=/usr/local/spark
spark.jars=
spark.logConf=true
spark.master=yarn
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS=proxyserver
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES=http://proxyserver:8089/proxy/application_1519904229414_0011
yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.web-proxy.address</name>
<value>proxyserver:8089</value>
</property>
<property>
<name>yarn.resourcemanager.fs.state-store.uri</name>
<value>/rmstate</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>resourcemanager:8088</value>
</property>
<property>
<name>yarn.timeline-service.generic-application-history.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>yarn.timeline-service.bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.timeline-service.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value>
</property>
<property>
<name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/app-logs</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>4.1</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>resourcemanager:8031</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>resourcemanager</value>
</property>
<property>
<name>yarn.timeline-service.hostname</name>
<value>historyserver</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://historyserver:8188/applicationhistory/logs/</value>
</property>
<property>
<name>yarn.resourcemanager.bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>resourcemanager:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>resourcemanager:8032</value>
</property>
</configuration>SW hadoop-3.0.0 Spark 2.3.0 (git revision a0d7949896) built for Hadoop 2.7.3 Nodes jupyter proxyserver historyserver resourcemanager nodemanager ( http://9d581cfb391f:8042 ) Config of sparkcontext spark.app.id=application_1519904229414_0011 spark.app.name=Spark shell spark.driver.appUIAddress= http://jupyter:4040 spark.driver.extraJavaOptions=-Dlog4j.configuration= file:/tmp/driver_log4j.properties spark.driver.host=jupyter spark.driver.port=35813 spark.executor.extraJavaOptions=-Dlog4j.configuration= file:/tmp/driver_log4j.properties spark.executor.id=driver spark.home=/usr/local/spark spark.jars= spark.logConf=true spark.master=yarn spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS=proxyserver spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES= http://proxyserver:8089/proxy/application_1519904229414_0011 yarn-site.xml <configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.web-proxy.address</name> <value>proxyserver:8089</value> </property> <property> <name>yarn.resourcemanager.fs.state-store.uri</name> <value>/rmstate</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>resourcemanager:8088</value> </property> <property> <name>yarn.timeline-service.generic-application-history.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.recovery.enabled</name> <value>true</value> </property> <property> <name>yarn.nodemanager.bind-host</name> <value>0.0.0.0</value> </property> <property> <name>yarn.timeline-service.bind-host</name> <value>0.0.0.0</value> </property> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <property> <name>yarn.timeline-service.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.store.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value> </property> <property> <name>yarn.resourcemanager.system-metrics-publisher.enabled</name> <value>true</value> </property> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property> <property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>4.1</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>resourcemanager:8031</value> </property> <property> <name>yarn.resourcemanager.hostname</name> <value>resourcemanager</value> </property> <property> <name>yarn.timeline-service.hostname</name> <value>historyserver</value> </property> <property> <name>yarn.log.server.url</name> <value> http://historyserver:8188/applicationhistory/logs/ </value> </property> <property> <name>yarn.resourcemanager.bind-host</name> <value>0.0.0.0</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>resourcemanager:8030</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>resourcemanager:8032</value> </property> </configuration>
Description
Access from Resourcemanager to Webui in a distributed environment results in a redirect loop. It seems webui is not recognizing request is coming from resourcemanager proxy.
Situation
Spark application running on jupyter
link in resourcemanger:8088 (behind Trackingui->Applicationmaster)
http://proxyserver:8089/proxy/application_1519904229414_0011/
Response of webui on jupyter:4040 (redirects to proxyserver)
<html>
<head>
{{ <title> Moved </title> </head> <body> <h1> Moved </h1> <div> Content has moved <a href="http://proxyserver:8089/proxy/application_1519904229414_0011/">here</a> </div> }}
</body>
</html>
log output of proxyserver (results in redirect)
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,905 DEBUG strategy.ExecuteProduceConsume: EPC Prod/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@5c4d391a producing
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,908 DEBUG protocol.RequestAuthCache: Auth cache not set in the context
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,908 DEBUG http.wire: >> "[\r][\n]"
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,908 DEBUG http.headers: >> GET / HTTP/1.1
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,908 DEBUG http.headers: >> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,908 DEBUG http.headers: >> Host: jupyter:4040
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG http.headers: << Server: Jetty(9.3.z-SNAPSHOT)
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG client.DefaultRedirectStrategy: Redirect requested to location 'http://proxyserver:8089/proxy/application_1519904229414_0001/'
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG http.wire: << " Moved[\n]"
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG http.wire: << " Content has moved [\n]"
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG http.wire: << " <a href="http://proxyserver:8089/proxy/application_1519904229414_0001/">here</a>[\n]"
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG conn.BasicClientConnectionManager: Connection can be kept alive indefinitely
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,911 DEBUG conn.DefaultClientConnectionOperator: Connecting to proxyserver:8089
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,912 DEBUG io.ManagedSelector: Queued change org.eclipse.jetty.io.ManagedSelector$Accept@5498de79 on org.eclipse.jetty.io.ManagedSelector@558bdf1f id=0 keys=63 selected=0
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,912 DEBUG io.ManagedSelector: Selector loop woken up from select, 0/63 selected
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,913 DEBUG strategy.ExecuteProduceConsume: EPC Pend/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@5c4d391a dispatch
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,913 DEBUG thread.QueuedThreadPool: queue EPC Pend/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@5c4d391a
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,913 DEBUG io.SelectChannelEndPoint: changeInterests p=false 0->1 for SelectChannelEndPoint@77d3aac7{/10.0.1.17:48699<>8089,Open,in,out,FI,,0/10000,HttpConnection@7d134d7f}
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,914 DEBUG conn.DefaultClientConnection: Sending request: GET /proxy/application_1519904229414_0001/ HTTP/1.1
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,914 DEBUG http.wire: >> "Accept-Encoding: gzip, deflate[\r][\n]"
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,914 DEBUG http.wire: >> "Host: proxyserver:8089[\r][\n]"
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,914 DEBUG http.headers: >> GET /proxy/application_1519904229414_0001/ HTTP/1.1
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG io.FillInterest: FillInterest@301a7eba{true,AC.ReadCB@7d134d7f{HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7
{/10.0.1.17:48699<->8089,Open,in,out,FI,-,2/10000,HttpConnection@7d134d7f} {io=1/0,kio=1,kro=1}][p=HttpParser\{s=START,0 of 0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=0,c=false,a=IDLE,uri=null}]}} fillable AC.ReadCB@7d134d7f{HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7{/10.0.1.17:48699<->8089,Open,in,out,FI,-,2/10000,HttpConnection@7d134d7f}{io=1/0,kio=1,kro=1}][p=HttpParser\{s=START,0 of 0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=0,c=false,a=IDLE,uri=null}]}
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG io.ChannelEndPoint: filled 426 SelectChannelEndPoint@77d3aac7{/10.0.1.17:48699<>8089,Open,in,out,,-,2/10000,HttpConnection@7d134d7f}
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG server.HttpConnection: HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7{/10.0.1.17:48699<
][p=HttpParser\{s=START,0 of 0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=0,c=false,a=IDLE,uri=null}] filled 426 HeapByteBuffer@5b3c5d02[p=0,l=426,c=8192,r=426]={<<<GET /proxy/applic... Keep-Alive\r\n\r\n>>>Accept-Language: ...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG http.HttpParser: SPACE2 --> REQUEST_VERSION
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG http.HttpParser: HEADER_IN_VALUE --> HEADER
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG http.HttpParser: HEADER_IN_VALUE --> HEADER
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG http.HttpParser: HEADER --> HEADER_VALUE
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,915 DEBUG server.HttpConnection: HttpConnection@7d134d7f[SelectChannelEndPoint@77d3aac7{/10.0.1.17:48699<>8089,Open,in,out,,-,0/10000,HttpConnection@7d134d7f}
][p=HttpParser\{s=END,0 of 0},g=HttpGenerator@17fc5588\{s=START},c=HttpChannelOverHttp@54440a5\{r=1,c=false,a=IDLE,uri=//proxyserver:8089/proxy/application_1519904229414_0001/}] parsed true HttpParser{s=END,0 of 0}
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,916 DEBUG server.session: session=null
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,916 DEBUG servlet.ServletHandler: call filter static_user_filter
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,918 DEBUG io.ManagedSelector: Selector loop waiting on select
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,929 DEBUG webproxy.WebAppProxyServlet: local InetAddress for proxy host: proxyserver/10.0.1.17
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,930 DEBUG conn.BasicClientConnectionManager: Get connection for route proxyserver/10.0.1.17->{}->http://jupyter:4040
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,933 DEBUG protocol.RequestTargetAuthentication: Target auth state: UNCHALLENGED
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,934 DEBUG conn.DefaultClientConnection: Sending request: GET / HTTP/1.1
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,934 DEBUG http.wire: >> "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8[\r][\n]"
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,934 DEBUG http.wire: >> "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36[\r][\n]"
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,934 DEBUG http.wire: >> "Host: jupyter:4040[\r][\n]"
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,934 DEBUG http.wire: >> "Connection: Keep-Alive[\r][\n]"
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,934 DEBUG http.headers: >> Connection: Keep-Alive
March 1st 2018, 12:38:50.000 /hadoop_proxyserver.1.ftfomwpuh9pmpwm87aon122vl proxyserver stderr | 2018-03-01 11:38:49,937 DEBUG http.headers: << Location: http://proxyserver:8089/proxy/application_1519904229414_0001/