[AMBARI-12369] if audit to DB is made true for hdfs then namenode is not able to connect to mysql DB

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.1
• Component/s: ambari-server
• Assignee: Jonathan Hurley
• Reporter: Jonathan Hurley
• Created: 10/Jul/15 03:28
• Updated: 14/Jul/15 15:38
• Resolved: 10/Jul/15 15:44

Description

== STR ==
1. install ambari with minimum services (hdfs,yarn)
2. install ranger
3. enable audit to DB
4. restart all affected services

check namenode logs

Issue:
====

getting Error connecting audit database continuously in name node logs :

2015-07-09 12:26:08,403 ERROR destination.DBAuditDestination (DBAuditDestination.java:connect(195)) - Error connecting audit database. dbURL=jdbc:mysql://os-s11-3-deepak-xasecure-6-1.openstacklocal/ranger_audit, dbUser=rangerlogger
javax.persistence.PersistenceException: Exception [EclipseLink-4003] (Eclipse Persistence Services - 2.5.2.v20131113-a7346c6): org.eclipse.persistence.exceptions.DatabaseException
Exception Description: Configuration error.  Class [com.mysql.jdbc.Driver] not found.
        at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:766)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.getAbstractSession(EntityManagerFactoryDelegate.java:204)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.createEntityManagerImpl(EntityManagerFactoryDelegate.java:304)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:336)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:302)
        at org.apache.ranger.audit.dao.DaoManager.getEntityManager(DaoManager.java:48)
        at org.apache.ranger.audit.destination.DBAuditDestination.connect(DBAuditDestination.java:186)
        at org.apache.ranger.audit.destination.DBAuditDestination.getTransaction(DBAuditDestination.java:252)
        at org.apache.ranger.audit.destination.DBAuditDestination.beginTransaction(DBAuditDestination.java:261)
        at org.apache.ranger.audit.destination.DBAuditDestination.log(DBAuditDestination.java:84)
        at org.apache.ranger.audit.provider.BaseAuditHandler.logJSON(BaseAuditHandler.java:161)
        at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:882)
        at org.apache.ranger.audit.queue.AuditFileSpool.runDoAs(AuditFileSpool.java:830)
        at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:759)
        at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:757)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
        at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:765)
        at java.lang.Thread.run(Thread.java:745)
Caused by: Exception [EclipseLink-4003] (Eclipse Persistence Services - 2.5.2.v20131113-a7346c6): org.eclipse.persistence.exceptions.DatabaseException
Exception Description: Configuration error.  Class [com.mysql.jdbc.Driver] not found.
        at org.eclipse.persistence.exceptions.DatabaseException.configurationErrorClassNotFound(DatabaseException.java:89)
        at org.eclipse.persistence.sessions.DefaultConnector.loadDriverClass(DefaultConnector.java:267)
        at org.eclipse.persistence.sessions.DefaultConnector.connect(DefaultConnector.java:85)
        at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
        at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.setOrDetectDatasource(DatabaseSessionImpl.java:204)
        at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:741)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:239)
        at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:685)
        ... 19 more
2015-07-09 12:26:08,420 ERROR destination.DBAuditDestination (DBAuditDestination.java:getEntityManager(223)) - DBAuditDestination.getEntityManager(): failed
javax.persistence.PersistenceException: Exception [EclipseLink-4003] (Eclipse Persistence Services - 2.5.2.v20131113-a7346c6): org.eclipse.persistence.exceptions.DatabaseException
Exception Description: Configuration error.  Class [com.mysql.jdbc.Driver] not found.
        at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:766)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.getAbstractSession(EntityManagerFactoryDelegate.java:204)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.createEntityManagerImpl(EntityManagerFactoryDelegate.java:304)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:336)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:302)
        at org.apache.ranger.audit.dao.DaoManager.getEntityManager(DaoManager.java:48)
        at org.apache.ranger.audit.destination.DBAuditDestination.getEntityManager(DBAuditDestination.java:221)
        at org.apache.ranger.audit.destination.DBAuditDestination.getTransaction(DBAuditDestination.java:255)
        at org.apache.ranger.audit.destination.DBAuditDestination.beginTransaction(DBAuditDestination.java:261)
        at org.apache.ranger.audit.destination.DBAuditDestination.log(DBAuditDestination.java:84)
        at org.apache.ranger.audit.provider.BaseAuditHandler.logJSON(BaseAuditHandler.java:161)
        at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:882)
        at org.apache.ranger.audit.queue.AuditFileSpool.runDoAs(AuditFileSpool.java:830)
        at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:759)
        at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:757)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
        at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:765)
        at java.lang.Thread.run(Thread.java:745)
Caused by: Exception [EclipseLink-4003] (Eclipse Persistence Services - 2.5.2.v20131113-a7346c6): org.eclipse.persistence.exceptions.DatabaseException
Exception Description: Configuration error.  Class [com.mysql.jdbc.Driver] not found.
        at org.eclipse.persistence.exceptions.DatabaseException.configurationErrorClassNotFound(DatabaseException.java:89)
        at org.eclipse.persistence.sessions.DefaultConnector.loadDriverClass(DefaultConnector.java:267)
        at org.eclipse.persistence.sessions.DefaultConnector.connect(DefaultConnector.java:85)
        at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
        at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.setOrDetectDatasource(DatabaseSessionImpl.java:204)
        at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:741)
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:239)
        at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:685)
        ... 19 more

Analyzed this issue and pointed out this commit could be causing the issue. Basically, JAVA_JDBC_LIBS is not in HADOOP_CLASSPATH, hence name node won't be able to talk to audit DB.

Workaround for now:
In Ambari go to HDFS -> Configs -> "Advanced hadoop-env” -> hadoop-env template and enter the following line after JAVA_JDBC_LIBS is defined.

export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${JAVA_JDBC_LIBS}:

Issue Links

• links to: Reviewboard