Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-6350

Unable to restart all host-components on 110 node cluster

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.6.1
    • 1.6.1
    • ambari-web
    • None

    Description

      On a 110 node cluster I went to the hosts page and went into All Hosts > Hosts > Restart All Components. Following that I got an error dialog (image attached) and the below exceptions in the log

      21:03:20,017 ERROR [qtp1391464722-1188] AmbariJpaLocalTxnInterceptor:114 - [DETAILED ERROR] Rollback reason:
      Local Exception Stack:
      Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.4.0.v20120608-r11652): org.eclipse.persistence.exceptions.DatabaseException
      Internal Exception: java.sql.BatchUpdateException: Batch entry 0 INSERT INTO requestoperationlevel (operation_level_id, cluster_name, host_component_name, host_name, level_name, request_id, service_name) VALUES (2, 'Horton', NULL, 'horton-1.c.pramod-thangali.internal,horton-10.c.pramod-thangali.internal,horton-100.c.pramod-thangali.internal,horton-101.c.pramod-thangali.internal,horton-103.c.pramod-thangali.internal,horton-104.c.pramod-thangali.internal,horton-105.c.pramod-thangali.internal,horton-106.c.pramod-thangali.internal,horton-107.c.pramod-thangali.internal,horton-108.c.pramod-thangali.internal,horton-109.c.pramod-thangali.internal,horton-11.c.pramod-thangali.internal,horton-12.c.pramod-thangali.internal,horton-13.c.pramod-thangali.internal,horton-14.c.pramod-thangali.internal,horton-15.c.pramod-thangali.internal,horton-16.c.pramod-thangali.internal,horton-17.c.pramod-thangali.internal,horton-18.c.pramod-thangali.internal,horton-19.c.pramod-thangali.internal,horton-2.c.pramod-thangali.internal,horton-20.c.pramod-thangali.internal,horton-21.c.pramod-thangali.internal,horton-22.c.pramod-thangali.internal,horton-23.c.pramod-thangali.internal,horton-24.c.pramod-thangali.internal,horton-25.c.pramod-thangali.internal,horton-26.c.pramod-thangali.internal,horton-27.c.pramod-thangali.internal,horton-28.c.pramod-thangali.internal,horton-29.c.pramod-thangali.internal,horton-3.c.pramod-thangali.internal,horton-30.c.pramod-thangali.internal,horton-31.c.pramod-thangali.internal,horton-32.c.pramod-thangali.internal,horton-33.c.pramod-thangali.internal,horton-34.c.pramod-thangali.internal,horton-35.c.pramod-thangali.internal,horton-36.c.pramod-thangali.internal,horton-37.c.pramod-thangali.internal,horton-38.c.pramod-thangali.internal,horton-39.c.pramod-thangali.internal,horton-4.c.pramod-thangali.internal,horton-40.c.pramod-thangali.internal,horton-41.c.pramod-thangali.internal,horton-42.c.pramod-thangali.internal,horton-43.c.pramod-thangali.internal,horton-44.c.pramod-thangali.internal,horton-45.c.pramod-thangali.internal,horton-46.c.pramod-thangali.internal,horton-47.c.pramod-thangali.internal,horton-48.c.pramod-thangali.internal,horton-49.c.pramod-thangali.internal,horton-5.c.pramod-thangali.internal,horton-50.c.pramod-thangali.internal,horton-51.c.pramod-thangali.internal,horton-52.c.pramod-thangali.internal,horton-53.c.pramod-thangali.internal,horton-54.c.pramod-thangali.internal,horton-55.c.pramod-thangali.internal,horton-56.c.pramod-thangali.internal,horton-57.c.pramod-thangali.internal,horton-58.c.pramod-thangali.internal,horton-59.c.pramod-thangali.internal,horton-6.c.pramod-thangali.internal,horton-60.c.pramod-thangali.internal,horton-61.c.pramod-thangali.internal,horton-62.c.pramod-thangali.internal,horton-63.c.pramod-thangali.internal,horton-64.c.pramod-thangali.internal,horton-65.c.pramod-thangali.internal,horton-66.c.pramod-thangali.internal,horton-67.c.pramod-thangali.internal,horton-68.c.pramod-thangali.internal,horton-69.c.pramod-thangali.internal,horton-7.c.pramod-thangali.internal,horton-70.c.pramod-thangali.internal,horton-71.c.pramod-thangali.internal,horton-72.c.pramod-thangali.internal,horton-73.c.pramod-thangali.internal,horton-74.c.pramod-thangali.internal,horton-75.c.pramod-thangali.internal,horton-76.c.pramod-thangali.internal,horton-77.c.pramod-thangali.internal,horton-78.c.pramod-thangali.internal,horton-79.c.pramod-thangali.internal,horton-8.c.pramod-thangali.internal,horton-80.c.pramod-thangali.internal,horton-81.c.pramod-thangali.internal,horton-82.c.pramod-thangali.internal,horton-83.c.pramod-thangali.internal,horton-84.c.pramod-thangali.internal,horton-85.c.pramod-thangali.internal,horton-86.c.pramod-thangali.internal,horton-87.c.pramod-thangali.internal,horton-88.c.pramod-thangali.internal,horton-9.c.pramod-thangali.internal,horton-90.c.pramod-thangali.internal,horton-91.c.pramod-thangali.internal,horton-92.c.pramod-thangali.internal,horton-93.c.pramod-thangali.internal,horton-94.c.pramod-thangali.internal,horton-95.c.pramod-thangali.internal,horton-96.c.pramod-thangali.internal,horton-97.c.pramod-thangali.internal,horton-98.c.pramod-thangali.internal,horton-99.c.pramod-thangali.internal,horton-master-1.c.pramod-thangali.internal,horton-master-2.c.pramod-thangali.internal,horton-master-3.c.pramod-thangali.internal', 'Host', 25, NULL) was aborted.  Call getNextException to see the cause.
      Error Code: 0
      Call: INSERT INTO requestoperationlevel (operation_level_id, cluster_name, host_component_name, host_name, level_name, request_id, service_name) VALUES (?, ?, ?, ?, ?, ?, ?)
              bind => [7 parameters bound]
      Query: InsertObjectQuery(org.apache.ambari.server.orm.entities.RequestResourceFilterEntity@2c6ae575)
              at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:333)
              at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1501)
              at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeJDK12BatchStatement(DatabaseAccessor.java:875)
              at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:145)
              at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.appendCall(ParameterizedSQLBatchWritingMechanism.java:88)
              at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:571)
              at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:537)
              at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1800)
              at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:286)
              at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:207)
              at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:193)
              at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.insertObject(DatasourceCallQueryMechanism.java:342)
              at org.eclipse.persistence.internal.queries.StatementQueryMechanism.insertObject(StatementQueryMechanism.java:162)
              at org.eclipse.persistence.internal.queries.StatementQueryMechanism.insertObject(StatementQueryMechanism.java:177)
              at org.eclipse.persistence.internal.queries.DatabaseQueryMechanism.insertObjectForWrite(DatabaseQueryMechanism.java:471)
              at org.eclipse.persistence.queries.InsertObjectQuery.executeCommit(InsertObjectQuery.java:80)
              at org.eclipse.persistence.queries.InsertObjectQuery.executeCommitWithChangeSet(InsertObjectQuery.java:90)
              at org.eclipse.persistence.internal.queries.DatabaseQueryMechanism.executeWriteWithChangeSet(DatabaseQueryMechanism.java:286)
              at org.eclipse.persistence.queries.WriteObjectQuery.executeDatabaseQuery(WriteObjectQuery.java:58) 
              at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:852)
              at org.eclipse.persistence.queries.DatabaseQuery.executeInUnitOfWork(DatabaseQuery.java:751)
              at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWorkObjectLevelModifyQuery(ObjectLevelModifyQuery.java:108)
              at org.eclipse.persistence.queries.ObjectLevelModifyQuery.executeInUnitOfWork(ObjectLevelModifyQuery.java:85)
              at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2875)
              at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1602)
              at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1584)
              at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1535)
              at org.eclipse.persistence.internal.sessions.CommitManager.commitNewObjectsForClassWithChangeSet(CommitManager.java:224)
              at org.eclipse.persistence.internal.sessions.CommitManager.commitAllObjectsForClassWithChangeSet(CommitManager.java:191)
              at org.eclipse.persistence.internal.sessions.CommitManager.commitAllObjectsWithChangeSet(CommitManager.java:136)
              at org.eclipse.persistence.internal.sessions.AbstractSession.writeAllObjectsWithChangeSet(AbstractSession.java:3914)
              at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabase(UnitOfWorkImpl.java:1419)
              at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitToDatabase(RepeatableWriteUnitOfWork.java:634)
              at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitToDatabaseWithChangeSet(UnitOfWorkImpl.java:1509)
              at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:266)
              at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1147)
              at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commitInternal(EntityTransactionImpl.java:84)
              at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:63)
              at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:91)
              at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
              at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
              at org.apache.ambari.server.actionmanager.ActionDBAccessorImpl$$EnhancerByGuice$$ddd0d231.persistActions(<generated>)
              at org.apache.ambari.server.actionmanager.ActionManager.sendActions(ActionManager.java:95)
              at org.apache.ambari.server.actionmanager.ActionManager.sendActions(ActionManager.java:84)
              at org.apache.ambari.server.controller.AmbariManagementControllerImpl.createAction(AmbariManagementControllerImpl.java:2583)
              at org.apache.ambari.server.controller.internal.RequestResourceProvider$1.invoke(RequestResourceProvider.java:124)
              at org.apache.ambari.server.controller.internal.RequestResourceProvider$1.invoke(RequestResourceProvider.java:121)
              at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:237)
              at org.apache.ambari.server.controller.internal.RequestResourceProvider.createResources(RequestResourceProvider.java:121)
              at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:274)
              at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:75)
              at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36)
              at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
              at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
              at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:103)
              at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:72)
              at org.apache.ambari.server.api.services.RequestService.createRequests(RequestService.java:119)
              at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)
      

      Attachments

        1. Screen Shot 2014-06-29 at 2.11.30 PM.png
          433 kB
          Andrii Tkach
        2. scr.png
          26 kB
          Andrii Tkach
        3. AMBARI-6350.patch
          3 kB
          Andrii Tkach

        Activity

          People

            atkach Andrii Tkach
            atkach Andrii Tkach
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: