[RANGER-3163] Ranger Database deadlock when Service creation and user sync are running parallel

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0, 2.2.0
• Component/s: Ranger
• Assignee: Sailaja Polavarapu
• Reporter: Sailaja Polavarapu
• Created: 27/Jan/21 23:07
• Updated: 06/Jan/22 11:36
• Resolved: 23/Feb/21 20:59

Description

Following is the error and some observations:
Error :
2021-01-20 13:00:44,036 ERROR org.apache.ranger.rest.ServiceREST: createService(RangerService={id=

{null}

guid={null} isEnabled={true} createdBy={null} updatedBy={null} createTime={null} updateTime={null} version={1} name={cm_hdfs} displayName={cm_hdfs} type={hdfs} description={Hdfs repo} tagService={null} configs={tag.download.auth.users=

{hdfs}

password=LOG-REDACTED policy.download.auth.users={hdfs} hadoop.security.authentication={kerberos} hadoop.rpc.protection={authentication} fs.default.name={hdfs://dy-upg41-3.dy-upg41.root.hwx.site:8020} hadoop.security.authorization={true} username={hdfs} } policyVersion={null} policyUpdateTime={null} tagVersion={1} tagUpdateTime={null} }) failed

javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.7.7.v20200504-69f2c2b80d): org.eclipse.persistence.exceptions.DatabaseException

Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction

Error Code: 1213
Call: INSERT INTO x_portal_user (ADDED_BY_ID, CREATE_TIME, EMAIL, FIRST_NAME, LAST_NAME, LOGIN_ID, NOTES, OTHER_ATTRIBUTES, PASSWORD, PUB_SCR_NAME, STATUS, UPDATE_TIME, UPD_BY_ID, USER_SRC) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)  

 

Observation: 1. ** Service creation and user sync are running parallel, the issue is occurring when the number of users are more (around 1000) to sync. This issue leads to failure in cm_hdfs service creation along with migration failure. 

                       2. Everytime issue occurs at the time when user sync tries to update admin users in x_portal_user.

                       3. I have tried increasing the innodb_lock_wait_timeout time from 50(default) to 120 on the cluster dy-upg41-3.dy-upg41.root.hwx.site, then migrated the cluster. Still the same issue occurred. 

Issue Links

• relates to: RANGER-3576 service creation is failing intermittently due to DB unique key constraint violation