Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: hedwig-server
    • Labels:
      None

      Description

      currently there is only one single thread scheduler shared across topic manager, subscription manager and bookkeeper manager (also region manager). if enqueue speed is faster than dequeue speed for per topic, lots of operations would be submitted to this single thread scheduler, which cause lots of operations are pending in the executor to execute.

      we need to adapt a multiple thread executor of this case. and it would be better to separate executors between managers.

        Activity

        Hide
        Sijie Guo added a comment -

        attach a patch to replace single thread scheduler with multiple thread scheduler, which use bookkeeper's orderSafeScheduler and partition requests by topics.

        Show
        Sijie Guo added a comment - attach a patch to replace single thread scheduler with multiple thread scheduler, which use bookkeeper's orderSafeScheduler and partition requests by topics.
        Hide
        Hadoop QA added a comment -

        Testing JIRA BOOKKEEPER-527

        Patch BOOKKEEPER-527.diff downloaded at Sat Dec 29 08:52:06 UTC 2012

        ----------------------------

        +1 PATCH_APPLIES
        +1 CLEAN
        +1 RAW_PATCH_ANALYSIS
        . +1 the patch does not introduce any @author tags
        . +1 the patch does not introduce any tabs
        . +1 the patch does not introduce any trailing spaces
        . +1 the patch does not introduce any line longer than 120
        . +1 the patch does adds/modifies 10 testcase(s)
        +1 RAT
        . +1 the patch does not seem to introduce new RAT warnings
        +1 JAVADOC
        . +1 the patch does not seem to introduce new Javadoc warnings
        +1 COMPILE
        . +1 HEAD compiles
        . +1 patch compiles
        . +1 the patch does not seem to introduce new javac warnings
        +1 FINDBUGS
        . +1 the patch does not seem to introduce new Findbugs warnings
        +1 TESTS
        . Tests run: 757
        +1 DISTRO
        . +1 distro tarball builds with the patch

        ----------------------------
        +1 Overall result, good!, no -1s

        The full output of the test-patch run is available at

        . https://builds.apache.org/job/bookkeeper-trunk-precommit-build/179/

        Show
        Hadoop QA added a comment - Testing JIRA BOOKKEEPER-527 Patch BOOKKEEPER-527.diff downloaded at Sat Dec 29 08:52:06 UTC 2012 ---------------------------- +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 120 . +1 the patch does adds/modifies 10 testcase(s) +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 FINDBUGS . +1 the patch does not seem to introduce new Findbugs warnings +1 TESTS . Tests run: 757 +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- +1 Overall result, good!, no -1s The full output of the test-patch run is available at . https://builds.apache.org/job/bookkeeper-trunk-precommit-build/179/
        Hide
        Ivan Kelly added a comment -

        The TODO needs to be solved before committing, as not removing from the map will lead to a memory leak. That said, im not sure how to solve it.

        Show
        Ivan Kelly added a comment - The TODO needs to be solved before committing, as not removing from the map will lead to a memory leak. That said, im not sure how to solve it.
        Hide
        Sijie Guo added a comment -

        Ivan Kelly for the TODO item, it was not introduced in this jira. It is a problem already there, if I remembered correctly. it would be better to have a separated jira working on fixing it.

        Show
        Sijie Guo added a comment - Ivan Kelly for the TODO item, it was not introduced in this jira. It is a problem already there, if I remembered correctly. it would be better to have a separated jira working on fixing it.
        Hide
        Sijie Guo added a comment -

        remove it from 4.3.0, as no urgent need in 4.3.0

        Show
        Sijie Guo added a comment - remove it from 4.3.0, as no urgent need in 4.3.0

          People

          • Assignee:
            Sijie Guo
            Reporter:
            Sijie Guo
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development