[AMBARI-15831] DB exception when trying to add 700 hosts to cluster - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: 2.2.2
Fix Version/s: 2.2.2
Component/s: ambari-server
Labels:
None

Description

Had a working cluster with 900 nodes and everything was fine. Proceeded to add 700 nodes (with no external-ip) to the cluster - which registered fine. Added DataNode, NodeManager and ResourceManager to all 700 nodes and clicked on Deploy in UI. One of the calls failed and UI wizard was stuck, with the following in ambari-server.log

06 Apr 2016 17:30:45,172 ERROR [qtp-ambari-client-2713] AmbariJpaLocalTxnInterceptor:180 - [DETAILED ERROR] Rollback reason:
Local Exception Stack:
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "clusterhostmapping_pkey"
Error Code: 0
Call: INSERT INTO ClusterHostMapping (cluster_id, host_id) VALUES (?, ?)
        bind => [2 parameters bound]
        at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:900)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
        at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
        at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
        at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
        at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
        at org.apache.ambari.server.state.cluster.ClustersImpl$$EnhancerByGuice$$f4eca4b6.mapHostClusterEntities(<generated>)
        at org.apache.ambari.server.state.cluster.ClustersImpl.mapHostToCluster(ClustersImpl.java:592)
        at org.apache.ambari.server.state.cluster.ClustersImpl.mapHostToCluster(ClustersImpl.java:616)
        at org.apache.ambari.server.state.cluster.ClustersImpl.updateHostWithClusterAndAttributes(ClustersImpl.java:492)
        at org.apache.ambari.server.controller.internal.HostResourceProvider.createHosts(HostResourceProvider.java:509)
        at org.apache.ambari.server.controller.internal.HostResourceProvider$1.invoke(HostResourceProvider.java:197)
        at org.apache.ambari.server.controller.internal.HostResourceProvider$1.invoke(HostResourceProvider.java:194)
        at org.apache.ambari.server.controller.internal.AbstractResourceProvider.invokeWithRetry(AbstractResourceProvider.java:450)
        at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:273)
        at org.apache.ambari.server.controller.internal.HostResourceProvider.createResources(HostResourceProvider.java:194)
        at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:289)
        at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:76)
        at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36)
        at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
        at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:106)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:75)
        at org.apache.ambari.server.api.services.HostService.createHosts(HostService.java:114)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
.....
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "clusterhostmapping_pkey"
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892)
        ... 115 more
06 Apr 2016 17:30:45,175 ERROR [qtp-ambari-client-2713] AmbariJpaLocalTxnInterceptor:188 - [DETAILED ERROR] Internal exception (1) :
org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "clusterhostmapping_pkey"
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
        at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
        at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
        at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
        at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
        at org.apache.ambari.server.state.cluster.ClustersImpl$$EnhancerByGuice$$f4eca4b6.mapHostClusterEntities(<generated>)
        at org.apache.ambari.server.state.cluster.ClustersImpl.mapHostToCluster(ClustersImpl.java:592)
        at org.apache.ambari.server.state.cluster.ClustersImpl.mapHostToCluster(ClustersImpl.java:616)
        at org.apache.ambari.server.state.cluster.ClustersImpl.updateHostWithClusterAndAttributes(ClustersImpl.java:492)
        at org.apache.ambari.server.controller.internal.HostResourceProvider.createHosts(HostResourceProvider.java:509)
        at org.apache.ambari.server.controller.internal.HostResourceProvider$1.invoke(HostResourceProvider.java:197)
        at org.apache.ambari.server.controller.internal.HostResourceProvider$1.invoke(HostResourceProvider.java:194)
        at org.apache.ambari.server.controller.internal.AbstractResourceProvider.invokeWithRetry(AbstractResourceProvider.java:450)
        at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:273)
        at org.apache.ambari.server.controller.internal.HostResourceProvider.createResources(HostResourceProvider.java:194)
        at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:289)
        at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:76)
        at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36)
        at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)

06 Apr 2016 17:30:45,176 ERROR [qtp-ambari-client-2713] BaseManagementHandler:66 - Caught a runtime exception while attempting to create a resource: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "clusterhostmapping_pkey"
Error Code: 0
Call: INSERT INTO ClusterHostMapping (cluster_id, host_id) VALUES (?, ?)
        bind => [2 parameters bound]
javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "clusterhostmapping_pkey"
Error Code: 0
Call: INSERT INTO ClusterHostMapping (cluster_id, host_id) VALUES (?, ?)
        bind => [2 parameters bound]
        at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:159)
        at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
        at org.apache.ambari.server.state.cluster.ClustersImpl.mapHostToCluster(ClustersImpl.java:592)
        at org.apache.ambari.server.state.cluster.ClustersImpl.mapHostToCluster(ClustersImpl.java:616)
        at org.apache.ambari.server.state.cluster.ClustersImpl.updateHostWithClusterAndAttributes(ClustersImpl.java:492)
        at org.apache.ambari.server.controller.internal.HostResourceProvider.createHosts(HostResourceProvider.java:509)
        at org.apache.ambari.server.controller.internal.HostResourceProvider$1.invoke(HostResourceProvider.java:197)
        at org.apache.ambari.server.controller.internal.HostResourceProvider$1.invoke(HostResourceProvider.java:194)
        at org.apache.ambari.server.controller.internal.AbstractResourceProvider.invokeWithRetry(AbstractResourceProvider.java:450)
        at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:273)
        at org.apache.ambari.server.controller.internal.HostResourceProvider.createResources(HostResourceProvider.java:194)
        at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:289)
        at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:76)
        at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:36)
        at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:72)
        at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:135)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:106)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:75)
        at org.apache.ambari.server.api.services.HostService.createHosts(HostService.java:114)
....
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "clusterhostmapping_pkey"
Error Code: 0
Call: INSERT INTO ClusterHostMapping (cluster_id, host_id) VALUES (?, ?)
        bind => [2 parameters bound]
        at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:900)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
        at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
        at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
        at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
        at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
        at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
        ... 100 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "clusterhostmapping_pkey"
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
        at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892)
        ... 112 more

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

AMBARI-15831.patch
12/Apr/16 12:12
3 kB
Papirkovskyy Myroslav

Issue Links

links to

ReviewBoard

Activity

People

Assignee:: Papirkovskyy Myroslav

Reporter:: Papirkovskyy Myroslav

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 12/Apr/16 12:04

Updated:: 12/Apr/16 15:17

Resolved:: 12/Apr/16 12:20