[AMBARI-15454] Blueprint install using config_recommendation_strategy is not functional

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0, 2.2.1
• Fix Version/s: 2.4.0
• Component/s: ambari-server
• Flags: Patch
• Assignee: Shantanu Mundkur
• Reporter: Shantanu Mundkur
• Created: 17/Mar/16 05:47
• Updated: 14/Jul/16 18:03
• Resolved: 24/Mar/16 16:26

Description

Blueprint install using config_recommendation strategy seems to hang for a long time (couple of hours?) and ends up logging exceptions continually to ambari-server.log. At the same time many hundreds of directories are seen getting created under /var/run/ambari-server/stack-recommendations (I have seen above 800-900). If you keep it running eventually the cluster install seems to start but fails miserably at least during the start and some of it makes obvious that configuration recommendations were NOT applied.

14 Mar 2016 23:26:33,784 ERROR [pool-8-thread-1] TopologyManager:737 - TopologyManager.ConfigureClusterTask: An exception occurred while attempting to process cluster configs and set on cluster:
java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
at com.google.common.collect.Maps.filterKeys(Maps.java:2089)
at org.apache.ambari.server.controller.internal.BlueprintConfigurationProcessor.doFilterStackDefaults(BlueprintConfigurationProcessor.java:445)
at org.apache.ambari.server.controller.internal.BlueprintConfigurationProcessor.doRecommendConfigurations(BlueprintConfigurationProcessor.java:418)
at org.apache.ambari.server.controller.internal.BlueprintConfigurationProcessor.doUpdateForClusterCreate(BlueprintConfigurationProcessor.java:225)
at org.apache.ambari.server.topology.ClusterConfigurationRequest.process(ClusterConfigurationRequest.java:97)
at org.apache.ambari.server.topology.TopologyManager$ConfigureClusterTask.call(TopologyManager.java:735)
at org.apache.ambari.server.topology.TopologyManager$ConfigureClusterTask.call(TopologyManager.java:709)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
14 Mar 2016 23:26:33,784 INFO [pool-3-thread-1] AsyncCallableService:111 - Exception during task execution:
java.util.concurrent.ExecutionException: java.lang.Exception: java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.ambari.server.topology.AsyncCallableService.taskCompleted(AsyncCallableService.java:103)
at org.apache.ambari.server.topology.AsyncCallableService.call(AsyncCallableService.java:74)
at org.apache.ambari.server.topology.AsyncCallableService.call(AsyncCallableService.java:37)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: java.lang.NullPointerException
at org.apache.ambari.server.topology.TopologyManager$ConfigureClusterTask.call(TopologyManager.java:741)
at org.apache.ambari.server.topology.TopologyManager$ConfigureClusterTask.call(TopologyManager.java:709)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more
Caused by: java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
at com.google.common.collect.Maps.filterKeys(Maps.java:2089)
at org.apache.ambari.server.controller.internal.BlueprintConfigurationProcessor.doFilterStackDefaults(BlueprintConfigurationProcessor.java:445)
at org.apache.ambari.server.controller.internal.BlueprintConfigurationProcessor.doRecommendConfigurations(BlueprintConfigurationProcessor.java:418)
at org.apache.ambari.server.controller.internal.BlueprintConfigurationProcessor.doUpdateForClusterCreate(BlueprintConfigurationProcessor.java:225)
at org.apache.ambari.server.topology.ClusterConfigurationRequest.process(ClusterConfigurationRequest.java:97)
at org.apache.ambari.server.topology.TopologyManager$ConfigureClusterTask.call(TopologyManager.java:735)
... 7 more

15 Mar 2016 16:49:09,508 INFO [pool-2-thread-1] AsyncCallableService:111 - Exception during task execution:
java.util.concurrent.ExecutionException: java.lang.Exception: java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:206)
at org.apache.ambari.server.topology.AsyncCallableService.taskCompleted(AsyncCallableService.java:103)
at org.apache.ambari.server.topology.AsyncCallableService.call(AsyncCallableService.java:74)
at org.apache.ambari.server.topology.AsyncCallableService.call(AsyncCallableService.java:37)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: java.lang.NullPointerException
at org.apache.ambari.server.topology.TopologyManager$ConfigureClusterTask.call(TopologyManager.java:741)
at org.apache.ambari.server.topology.TopologyManager$ConfigureClusterTask.call(TopologyManager.java:709)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more
Caused by: java.lang.NullPointerException
at org.apache.ambari.server.controller.internal.BlueprintConfigurationProcessor.doReplaceProperties(BlueprintConfigurationProcessor.java:472)
at org.apache.ambari.server.controller.internal.BlueprintConfigurationProcessor.doRecommendConfigurations(BlueprintConfigurationProcessor.java:425)
at org.apache.ambari.server.controller.internal.BlueprintConfigurationProcessor.doUpdateForClusterCreate(BlueprintConfigurationProcessor.java:225)
at org.apache.ambari.server.topology.ClusterConfigurationRequest.process(ClusterConfigurationRequest.java:96)
at org.apache.ambari.server.topology.TopologyManager$ConfigureClusterTask.call(TopologyManager.java:735)
... 7 more

Issue Links

• links to: ReviewBoard