Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12937

DbNotificationListener unable to clean up old notification events

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.1, 1.3.0, 2.0.0, 2.1.0
    • Fix Version/s: 1.2.2, 1.3.0, 2.0.1, 2.1.0
    • Component/s: None
    • Labels:
      None

      Description

      There is a bug in ObjectStore, where we use pm.deletePersistent instead of pm.deletePersistentAll, which causes the persistenceManager to try and drop a org.datanucleus.store.rdbms.query.ForwardQueryResult instead of the appropriate associated org.apache.hadoop.hive.metastore.model.MNotificationLog.

      This results in an error that looks like this:

      Exception in thread "CleanerThread" org.datanucleus.api.jdo.exceptions.ClassNotPersistenceCapableException: The class "org.datanucleus.store.rdbms.query.ForwardQueryResult" is not persistable. This means that it either hasnt been enhanced, or that the enhanced version of the file is not in the CLASSPATH (or is hidden by an unenhanced version), or the Meta-Data/annotations for the class are not found.
      at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:380)
      at org.datanucleus.api.jdo.JDOPersistenceManager.jdoDeletePersistent(JDOPersistenceManager.java:807)
      at org.datanucleus.api.jdo.JDOPersistenceManager.deletePersistent(JDOPersistenceManager.java:820)
      at org.apache.hadoop.hive.metastore.ObjectStore.cleanNotificationEvents(ObjectStore.java:7149)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
      at com.sun.proxy.$Proxy0.cleanNotificationEvents(Unknown Source)
      at org.apache.hive.hcatalog.listener.DbNotificationListener$CleanerThread.run(DbNotificationListener.java:277)
      NestedThrowablesStackTrace:
      The class "org.datanucleus.store.rdbms.query.ForwardQueryResult" is not persistable. This means that it either hasnt been enhanced, or that the enhanced version of the file is not in the CLASSPATH (or is hidden by an unenhanced version), or the Meta-Data/annotations for the class are not found.
      org.datanucleus.exceptions.ClassNotPersistableException: The class "org.datanucleus.store.rdbms.query.ForwardQueryResult" is not persistable. This means that it either hasnt been enhanced, or that the enhanced version of the file is not in the CLASSPATH (or is hidden by an unenhanced version), or the Meta-Data/annotations for the class are not found.
      at org.datanucleus.ExecutionContextImpl.assertClassPersistable(ExecutionContextImpl.java:5698)
      at org.datanucleus.ExecutionContextImpl.deleteObjectInternal(ExecutionContextImpl.java:2495)
      at org.datanucleus.ExecutionContextImpl.deleteObjectWork(ExecutionContextImpl.java:2466)
      at org.datanucleus.ExecutionContextImpl.deleteObject(ExecutionContextImpl.java:2417)
      at org.datanucleus.ExecutionContextThreadedImpl.deleteObject(ExecutionContextThreadedImpl.java:245)
      at org.datanucleus.api.jdo.JDOPersistenceManager.jdoDeletePersistent(JDOPersistenceManager.java:802)
      at org.datanucleus.api.jdo.JDOPersistenceManager.deletePersistent(JDOPersistenceManager.java:820)
      at org.apache.hadoop.hive.metastore.ObjectStore.cleanNotificationEvents(ObjectStore.java:7149)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
      at com.sun.proxy.$Proxy0.cleanNotificationEvents(Unknown Source)
      at org.apache.hive.hcatalog.listener.DbNotificationListener$CleanerThread.run(DbNotificationListener.java:277)
      

      The end result of this bug is that users of DbNotificationListener will have an evergrowing number of notification events that are not cleaned up as they age. This is an easy enough fix, but shows that we have a lack of test coverage here.

        Attachments

        1. HIVE-12937.patch
          3 kB
          Sushanth Sowmyan

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              sushanth Sushanth Sowmyan Assign to me
              Reporter:
              sushanth Sushanth Sowmyan

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment