Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2.1, 4.3.0
    • Component/s: None
    • Labels:
      None

      Description

      ReadOnlyBookieTest hangs on shutdown. It occurs when I run bookie tests.

      1. 27376.txt
        11 kB
        Sijie Guo
      2. BOOKKEEPER-567.patch
        0.8 kB
        Sijie Guo

        Activity

        Hide
        Sijie Guo added a comment -

        attach the jstack.

        Show
        Sijie Guo added a comment - attach the jstack.
        Hide
        Sijie Guo added a comment -

        this race condition is triggered when SyncThread tried to shutdown bookie encountering NoWritableLedgerDirException. the sequence is:

        1) SyncThread -> transitionToReadOnlyMode -> triggerShutDown
        2) triggerShutdown starts a new thread to shutdown bookie. SyncThread joins to wait shutdown thread finished.
        3) Shutdown thread tried to call bookie#shutdown -> SyncThread#shutdown.
        4) SyncThread#shutdown checked flushing flag and found it was in flushing procedure. it doesn't interrupt SyncThread and just wait.

        Show
        Sijie Guo added a comment - this race condition is triggered when SyncThread tried to shutdown bookie encountering NoWritableLedgerDirException. the sequence is: 1) SyncThread -> transitionToReadOnlyMode -> triggerShutDown 2) triggerShutdown starts a new thread to shutdown bookie. SyncThread joins to wait shutdown thread finished. 3) Shutdown thread tried to call bookie#shutdown -> SyncThread#shutdown. 4) SyncThread#shutdown checked flushing flag and found it was in flushing procedure. it doesn't interrupt SyncThread and just wait.
        Hide
        Sijie Guo added a comment -

        attach a patch to clear flushing flag before trigger shutdown.

        Show
        Sijie Guo added a comment - attach a patch to clear flushing flag before trigger shutdown.
        Hide
        Hadoop QA added a comment -

        Testing JIRA BOOKKEEPER-567

        Patch BOOKKEEPER-567.patch downloaded at Sat Feb 9 09:41:13 UTC 2013

        ----------------------------

        +1 PATCH_APPLIES
        +1 CLEAN
        -1 RAW_PATCH_ANALYSIS
        . +1 the patch does not introduce any @author tags
        . +1 the patch does not introduce any tabs
        . +1 the patch does not introduce any trailing spaces
        . +1 the patch does not introduce any line longer than 120
        . -1 the patch does not add/modify any testcase
        +1 RAT
        . +1 the patch does not seem to introduce new RAT warnings
        +1 JAVADOC
        . +1 the patch does not seem to introduce new Javadoc warnings
        +1 COMPILE
        . +1 HEAD compiles
        . +1 patch compiles
        . +1 the patch does not seem to introduce new javac warnings
        +1 FINDBUGS
        . +1 the patch does not seem to introduce new Findbugs warnings
        +1 TESTS
        . Tests run: 816
        +1 DISTRO
        . +1 distro tarball builds with the patch

        ----------------------------
        -1 Overall result, please check the reported -1(s)

        The full output of the test-patch run is available at

        . https://builds.apache.org/job/bookkeeper-trunk-precommit-build/262/

        Show
        Hadoop QA added a comment - Testing JIRA BOOKKEEPER-567 Patch BOOKKEEPER-567.patch downloaded at Sat Feb 9 09:41:13 UTC 2013 ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 120 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 FINDBUGS . +1 the patch does not seem to introduce new Findbugs warnings +1 TESTS . Tests run: 816 +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/bookkeeper-trunk-precommit-build/262/
        Hide
        Ivan Kelly added a comment -

        +1

        Fix looks good to me. I just ran into this on the 4.2.1 branch also, so committing to both.

        Show
        Ivan Kelly added a comment - +1 Fix looks good to me. I just ran into this on the 4.2.1 branch also, so committing to both.
        Hide
        Ivan Kelly added a comment -

        Committed as r1447772(trunk), r1447774(branch-4.2)

        Show
        Ivan Kelly added a comment - Committed as r1447772(trunk), r1447774(branch-4.2)
        Hide
        Hudson added a comment -

        Integrated in bookkeeper-trunk #111 (See https://builds.apache.org/job/bookkeeper-trunk/111/)
        BOOKKEEPER-567: ReadOnlyBookieTest hangs on shutdown (sijie via ivank) (Revision 1447772)

        Result = SUCCESS
        ivank :
        Files :

        • /zookeeper/bookkeeper/trunk/CHANGES.txt
        • /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
        Show
        Hudson added a comment - Integrated in bookkeeper-trunk #111 (See https://builds.apache.org/job/bookkeeper-trunk/111/ ) BOOKKEEPER-567 : ReadOnlyBookieTest hangs on shutdown (sijie via ivank) (Revision 1447772) Result = SUCCESS ivank : Files : /zookeeper/bookkeeper/trunk/CHANGES.txt /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java

          People

          • Assignee:
            Sijie Guo
            Reporter:
            Sijie Guo
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development