Uploaded image for project: 'Metron'
  1. Metron
  2. METRON-2088 Support HDP 3.1.0
  3. METRON-2250

Missing services in HDP 3.1 metron mpack and installer stuck

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Done
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      We missed this during our full dev deployment because we use blueprints, which seems to bypass the service_advisor mechanism in Ambari.

       

      If you uninstall Metron and attempt to reinstall the service (or do any fresh install for that matter), the service recommendations will fail due to missing hostnames in the services list. The following stack traces appear in the Ambari server logs.

       

      Error occured in stack advisor.
      Error details: list index out of range
      2019-09-03 20:42:10,911 INFO [ambari-client-thread-34] StackAdvisorRunner:167 - Advisor script stderr: Traceback (most recent call last):
       File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 190, in <module>
       main(sys.argv)
       File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 122, in main
       result = stackAdvisor.validateComponentLayout(services, hosts)
       File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 1072, in validateComponentLayout
       validationItems = self.getComponentLayoutValidations(services, hosts)
       File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 1095, in getComponentLayoutValidations
       items.extend(serviceAdvisor.getServiceComponentLayoutValidations(services, hosts))
       File "/var/lib/ambari-server/resources/common-services/METRON/0.7.2/service_advisor.py", line 46, in getServiceComponentLayoutValidations
       metronParsersHost = self.getHosts(componentsList, "METRON_PARSERS")[0]
      IndexError: list index out of range
      2019-09-03 20:42:10,912 WARN [ambari-client-thread-34] ValidationResourceProvider:132 - Error occurred during validation
      org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: Stack Advisor reported an error. Exit Code: 2. Error: IndexError: list index out of range
      StdOut file: /var/run/ambari-server/stack-recommendations/7/stackadvisor.out
      StdErr file: /var/run/ambari-server/stack-recommendations/7/stackadvisor.err
       at org.apache.ambari.server.api.services.stackadvisor.StackAdvisorRunner.processLogs(StackAdvisorRunner.java:149)
       at org.apache.ambari.server.api.services.stackadvisor.StackAdvisorRunner.runScript(StackAdvisorRunner.java:89)
       at org.apache.ambari.server.api.services.stackadvisor.commands.StackAdvisorCommand.invoke(StackAdvisorCommand.java:314)
       at org.apache.ambari.server.api.services.stackadvisor.StackAdvisorHelper.validate(StackAdvisorHelper.java:94)
       at org.apache.ambari.server.controller.internal.ValidationResourceProvider.createResources(ValidationResourceProvider.java:127)
       at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:296)
       at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:97)
       at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:50)
       at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:68)
       at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:144)
       at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:163)
       at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:127)
       at org.apache.ambari.server.api.services.ValidationService.getValidation(ValidationService.java:59)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
       at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
      at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
       at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
       at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
       at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
       at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
       at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
       at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
       at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
       at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
       at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
       at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
       at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
       at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
       at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
       at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
       at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:291)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
       at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
       at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
       at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
       at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
       at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
       at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFilter(AmbariDelegatingAuthenticationFilter.java:135)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
       at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:95)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
       at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
       at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
       at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
       at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
       at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
       at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
       at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
       at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
       at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
       at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:73)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
       at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:53)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
       at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:130)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
       at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:51)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
       at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:690)
       at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:221)
       at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:210)
       at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:140)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.Server.handle(Server.java:503)
       at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
       at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
       at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
       at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
       at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
       at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
       at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
       at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
       at java.lang.Thread.run(Thread.java:745)
      

       

       

      The recommended hosts are not something we own, afaik, so it's highly likely that this is a bug in Ambari.

       

        Attachments

          Activity

            People

            • Assignee:
              mohandv Mohan Venkateshaiah
              Reporter:
              mmiklavcic Michael Miklavcic
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 3h 10m
                3h 10m