[AMBARI-17036] Getting JMX Protocol Values On Large Cluster Takes Too Long

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.4.0
• Component/s: ambari-server
• Assignee: Jonathan Hurley
• Reporter: Jonathan Hurley
• Created: 03/Jun/16 19:56
• Updated: 14/Jun/16 20:31
• Resolved: 14/Jun/16 20:31

Description

When retrieving JMX properties on large clusters, if there are JMX endpoints which are unreachable, the URI protocol value is never cached. As a result, multiple attempts are made to get the JMX protocol. Normally, this would be fine, except that retrieving the configuration is using the ClusterResourceProvider, which in turn, builds a ClusterHealthReport.

	at org.apache.ambari.server.orm.dao.ServiceDesiredStateDAO.findByPK(ServiceDesiredStateDAO.java:41)
	at org.apache.ambari.server.orm.dao.ServiceDesiredStateDAO$$EnhancerByGuice$$d1ec71cf.CGLIB$findByPK$1(<generated>)
	at org.apache.ambari.server.orm.dao.ServiceDesiredStateDAO$$EnhancerByGuice$$d1ec71cf$$FastClassByGuice$$9cad416f.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
	at org.apache.ambari.server.orm.AmbariLocalSessionInterceptor.invoke(AmbariLocalSessionInterceptor.java:43)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
	at org.apache.ambari.server.orm.dao.ServiceDesiredStateDAO$$EnhancerByGuice$$d1ec71cf.findByPK(<generated>)
	at org.apache.ambari.server.state.ServiceImpl.getServiceDesiredStateEntity(ServiceImpl.java:759)
	at org.apache.ambari.server.state.ServiceImpl.getMaintenanceState(ServiceImpl.java:732)
	at org.apache.ambari.server.controller.MaintenanceStateHelper.getEffectiveState(MaintenanceStateHelper.java:216)
	at org.apache.ambari.server.controller.MaintenanceStateHelper.getEffectiveState(MaintenanceStateHelper.java:180)
	at org.apache.ambari.server.controller.MaintenanceStateHelper.getEffectiveState(MaintenanceStateHelper.java:194)
	at org.apache.ambari.server.state.cluster.ClusterImpl.getClusterHealthReport(ClusterImpl.java:3136)
	at org.apache.ambari.server.state.cluster.ClusterImpl.convertToResponse(ClusterImpl.java:2084)
	at org.apache.ambari.server.controller.AmbariManagementControllerImpl.getClusters(AmbariManagementControllerImpl.java:950)
	at org.apache.ambari.server.controller.AmbariManagementControllerImpl.getClusters(AmbariManagementControllerImpl.java:3082)
	at org.apache.ambari.server.controller.internal.ClusterResourceProvider$1.invoke(ClusterResourceProvider.java:202)
	at org.apache.ambari.server.controller.internal.ClusterResourceProvider$1.invoke(ClusterResourceProvider.java:199)
	at org.apache.ambari.server.controller.internal.AbstractResourceProvider.getResources(AbstractResourceProvider.java:302)
	at org.apache.ambari.server.controller.internal.ClusterResourceProvider.getResources(ClusterResourceProvider.java:199)
	at org.apache.ambari.server.controller.internal.AbstractProviderModule.getDesiredConfigVersion(AbstractProviderModule.java:954)
	at org.apache.ambari.server.controller.internal.AbstractProviderModule.getJMXProtocol(AbstractProviderModule.java:1154)
	at org.apache.ambari.server.controller.jmx.JMXPropertyProvider.getJMXProtocol(JMXPropertyProvider.java:402)
	at org.apache.ambari.server.controller.jmx.JMXPropertyProvider.populateResource(JMXPropertyProvider.java:221)
	at org.apache.ambari.server.controller.metrics.ThreadPoolEnabledPropertyProvider$1.call(ThreadPoolEnabledPropertyProvider.java:222)
	at org.apache.ambari.server.controller.metrics.ThreadPoolEnabledPropertyProvider$1.call(ThreadPoolEnabledPropertyProvider.java:219)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Issue Links

• links to: Reviewboard