Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-6722

Memory leak after updating to RocksDB 7

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 1.3.0
    • 1.3.0
    • None

    Description

      After HDDS-6456 updated RocksDB version from 6.25.3 to 7.0.4, we started to experience OOM at Ozone Manager very quickly.

      In our test environment, the OM memory stayed at around 3GB. But after that, OM memory quickly increase to over 100GB and crashed within 2 hours.

      After some investigation and discussion with the RocksDB developers (https://github.com/facebook/rocksdb/issues/9962) we were able to isolate the issue to this particular breaking change: https://github.com/facebook/rocksdb/commit/99d86252b

      Prior to the change, RocksDB 6 automatically reclaims native memory when the corresponding Java object is garbage collected. After the change (RocksDB 7), applications must proactively close the objects unused.

      The biggest offenders are the RocksIterator. But there are many others. I'll file jira to clean up their usage. Unfortunately, SonarCloud doesn't seem to detect these unclosed objects.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            duongnguyen Duong
            weichiu Wei-Chiu Chuang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment