Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-26376

Hive Metastore connection leak (OOM Error)

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.2
    • None
    • Metastore

    Description

      Hive version:3.1.2

      Hive metastore heap size is 14GB, Memory Leak is happening after 4-5 days, hive meta-store throwing error with OOM.

      If we disable the configuration, the memory leak disappears.

      In the case of, Heap dump size 3.5GB, a large number of filesystem objects(> 9k instances) are being retained. It's occupying most of the heap space. Added snapshot from the eclipse MAT.

      Bellow are part of the stack trace for OOM error:

      at org.apache.hadoop.hive.common.FileUtils.getFileStatusOrNull(Lorg/apache/hadoop/fs/FileSystem;Lorg/apache/hadoop/fs/Path;)Lorg/apache/hadoop/fs/FileStatus; (FileUtils.java:801)
        at org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.checkPermissions(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/fs/Path;Ljava/util/EnumSet;)V (StorageBasedAuthorizationProvider.java:371)
        at org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(Lorg/apache/hadoop/fs/Path;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;)V (StorageBasedAuthorizationProvider.java:346)
        at org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider.authorize(Lorg/apache/hadoop/hive/metastore/api/Database;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;[Lorg/apache/hadoop/hive/ql/security/authorization/Privilege;)V (StorageBasedAuthorizationProvider.java:154)
        at org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener.authorizeReadDatabase(Lorg/apache/hadoop/hive/metastore/events/PreReadDatabaseEvent;)V (AuthorizationPreEventListener.java:208)
        at org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener.onEvent(Lorg/apache/hadoop/hive/metastore/events/PreEventContext;)V (AuthorizationPreEventListener.java:153)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(Lorg/apache/hadoop/hive/metastore/events/PreEventContext;)V (HiveMetaStore.java:3221)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(Ljava/lang/String;)Lorg/apache/hadoop/hive/metastore/api/Database; (HiveMetaStore.java:1352)

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zabetak Stamatis Zampetakis Assign to me
            RANith Ranith Sardar

            Dates

              Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 0.5h
              0.5h

              Slack

                Issue deployment