Bookkeeper
  1. Bookkeeper
  2. BOOKKEEPER-529

stopServingSubscriber in delivery manager should remove stub callbacks in ReadAheadCache

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2.0
    • Component/s: hedwig-server
    • Labels:
      None

      Description

      Currently each subscriber would insert stub callback to wait newly published messages or scanning result. for waiting scan result, it was OK. the callback would be triggered and removed after scan callback arrived. but if it was wait newly published, it would be a problem. if sub/closesub/sub become frequent, closesub doesn't remove the installed callback, so the stub callbacks are accumulated, which cause the memory increased, finally OOM.

      it would be better to remove its installed stub callback when closesub.

      1. BOOKKEEPER-529.diff
        21 kB
        Sijie Guo
      2. BOOKKEEPER-529.diff
        20 kB
        Sijie Guo

        Issue Links

          Activity

          Ivan Kelly made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          Integrated in bookkeeper-trunk2 #47 (See https://builds.apache.org/job/bookkeeper-trunk2/47/)
          BOOKKEEPER-529: stopServingSubscriber in delivery manager should remove stub callbacks in ReadAheadCache (sijie via ivank) (Revision 1430954)

          Result = SUCCESS
          ivank :
          Files :

          • /zookeeper/bookkeeper/trunk/CHANGES.txt
          • /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/delivery/FIFODeliveryManager.java
          • /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/persistence/CacheValue.java
          • /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/persistence/CancelScanRequest.java
          • /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/persistence/ReadAheadCache.java
          • /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/persistence/ScanCallbackWithContext.java
          • /zookeeper/bookkeeper/trunk/hedwig-server/src/test/java/org/apache/hedwig/client/TestPubSubClient.java
          Show
          Hudson added a comment - Integrated in bookkeeper-trunk2 #47 (See https://builds.apache.org/job/bookkeeper-trunk2/47/ ) BOOKKEEPER-529 : stopServingSubscriber in delivery manager should remove stub callbacks in ReadAheadCache (sijie via ivank) (Revision 1430954) Result = SUCCESS ivank : Files : /zookeeper/bookkeeper/trunk/CHANGES.txt /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/delivery/FIFODeliveryManager.java /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/persistence/CacheValue.java /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/persistence/CancelScanRequest.java /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/persistence/ReadAheadCache.java /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/persistence/ScanCallbackWithContext.java /zookeeper/bookkeeper/trunk/hedwig-server/src/test/java/org/apache/hedwig/client/TestPubSubClient.java
          Ivan Kelly made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Ivan Kelly added a comment -

          Committed as r1430954. Thanks Sijie.

          Show
          Ivan Kelly added a comment - Committed as r1430954. Thanks Sijie.
          Hide
          Ivan Kelly added a comment -

          Looks good to me +1. Committing.

          Show
          Ivan Kelly added a comment - Looks good to me +1. Committing.
          Hide
          Hadoop QA added a comment -

          Testing JIRA BOOKKEEPER-529

          Patch BOOKKEEPER-529.diff downloaded at Wed Jan 9 07:11:24 UTC 2013

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          +1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 120
          . +1 the patch does adds/modifies 1 testcase(s)
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          +1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . +1 the patch does not seem to introduce new javac warnings
          +1 FINDBUGS
          . +1 the patch does not seem to introduce new Findbugs warnings
          +1 TESTS
          . Tests run: 783
          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          +1 Overall result, good!, no -1s

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/bookkeeper-trunk-precommit-build/222/

          Show
          Hadoop QA added a comment - Testing JIRA BOOKKEEPER-529 Patch BOOKKEEPER-529.diff downloaded at Wed Jan 9 07:11:24 UTC 2013 ---------------------------- +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 120 . +1 the patch does adds/modifies 1 testcase(s) +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 FINDBUGS . +1 the patch does not seem to introduce new Findbugs warnings +1 TESTS . Tests run: 783 +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- +1 Overall result, good!, no -1s The full output of the test-patch run is available at . https://builds.apache.org/job/bookkeeper-trunk-precommit-build/222/
          Sijie Guo made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Sijie Guo made changes -
          Attachment BOOKKEEPER-529.diff [ 12563896 ]
          Hide
          Sijie Guo added a comment -

          attach a new patch rebased to latest trunk.

          Show
          Sijie Guo added a comment - attach a new patch rebased to latest trunk.
          Sijie Guo made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Sijie Guo added a comment -

          move 4.2.0 release, since it would resolve BOOKKEEPER-539's issue.

          Show
          Sijie Guo added a comment - move 4.2.0 release, since it would resolve BOOKKEEPER-539 's issue.
          Sijie Guo made changes -
          Fix Version/s 4.2.0 [ 12320244 ]
          Fix Version/s 4.3.0 [ 12323267 ]
          Sijie Guo made changes -
          Link This issue blocks BOOKKEEPER-539 [ BOOKKEEPER-539 ]
          Hide
          Fangmin Lv added a comment -

          Good patch, lgtm +1, with this patch we can obviously reduce memory leak, it has been verified on our production environment(only 8G memory was used while others 12G after two days running).

          Show
          Fangmin Lv added a comment - Good patch, lgtm +1, with this patch we can obviously reduce memory leak, it has been verified on our production environment(only 8G memory was used while others 12G after two days running).
          Hide
          Hadoop QA added a comment -

          Testing JIRA BOOKKEEPER-529

          Patch BOOKKEEPER-529.diff downloaded at Thu Jan 3 06:51:12 UTC 2013

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          +1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 120
          . +1 the patch does adds/modifies 1 testcase(s)
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          +1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . +1 the patch does not seem to introduce new javac warnings
          +1 FINDBUGS
          . +1 the patch does not seem to introduce new Findbugs warnings
          +1 TESTS
          . Tests run: 761
          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          +1 Overall result, good!, no -1s

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/bookkeeper-trunk-precommit-build/190/

          Show
          Hadoop QA added a comment - Testing JIRA BOOKKEEPER-529 Patch BOOKKEEPER-529.diff downloaded at Thu Jan 3 06:51:12 UTC 2013 ---------------------------- +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 120 . +1 the patch does adds/modifies 1 testcase(s) +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 FINDBUGS . +1 the patch does not seem to introduce new Findbugs warnings +1 TESTS . Tests run: 761 +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- +1 Overall result, good!, no -1s The full output of the test-patch run is available at . https://builds.apache.org/job/bookkeeper-trunk-precommit-build/190/
          Sijie Guo made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Sijie Guo [ hustlmsp ]
          Sijie Guo made changes -
          Field Original Value New Value
          Attachment BOOKKEEPER-529.diff [ 12563030 ]
          Sijie Guo created issue -

            People

            • Assignee:
              Sijie Guo
              Reporter:
              Sijie Guo
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development