ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-486

Improve bookie performance for large number of ledgers

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: contrib-bookkeeper
    • Labels:
      None

      Description

      If we write simultaneously to a large number of ledgers on a bookie, then performance drops significantly due to more seeks on the ledger device. In such cases, we should be clustering ledgers into files to reduce the number of seeks, and performing sequential writes on each file. Clustering ledgers will impact read performance, so we would to have a knob to control such a feature.

      1. ZOOKEEPER-486.patch
        91 kB
        Flavio Junqueira
      2. ZOOKEEPER-486.patch
        92 kB
        Benjamin Reed
      3. ZOOKEEPER-486.patch
        63 kB
        Benjamin Reed
      4. ZOOKEEPER-486.patch
        99 kB
        Benjamin Reed
      5. ZOOKEEPER-486.patch
        100 kB
        Benjamin Reed

        Activity

        Hide
        Utkarsh Srivastava added a comment -

        Changes to handle this are included in the patch for ZOOKEEPER-507.

        Show
        Utkarsh Srivastava added a comment - Changes to handle this are included in the patch for ZOOKEEPER-507 .
        Hide
        Flavio Junqueira added a comment -

        Separated the bookie part of patch ZOOKEEPER-507. There is one test failing because a bookie is apparently failing to handle a missing ledger.

        Show
        Flavio Junqueira added a comment - Separated the bookie part of patch ZOOKEEPER-507 . There is one test failing because a bookie is apparently failing to handle a missing ledger.
        Hide
        Flavio Junqueira added a comment -

        Here is the log output for the failing test:

        2009-12-02 09:46:22,952 - INFO  - [main:BookieClientTest@198] - Starting no ledger test
        2009-12-02 09:46:22,956 - INFO  - [Thread-4:BookieClientTest$1@75] - Capacity 16, 0
        2009-12-02 09:46:22,957 - INFO  - [Thread-4:BookieClientTest$1@83] - Received 0
        2009-12-02 09:46:23,001 - ERROR - [SyncThread:Bookie$LastLogMark@295] - Problems writing to /var/folders/iq/iqmyL+XiGjqE2AjRkKTe-E+++TI/-Tmp-/bookie2550661167481866393test/lastMark
        java.io.FileNotFoundException: /var/folders/iq/iqmyL+XiGjqE2AjRkKTe-E+++TI/-Tmp-/bookie2550661167481866393test/lastMark (No such file or directory)
        	at java.io.FileOutputStream.open(Native Method)
        	at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
        	at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
        	at org.apache.bookkeeper.bookie.Bookie$LastLogMark.rollLog(Bookie.java:290)
        	at org.apache.bookkeeper.bookie.Bookie$SyncThread.run(Bookie.java:116)
        2009-12-02 09:46:23,002 - FATAL - [Thread-3:Bookie@375] - Bookie thread exiting
        java.lang.InterruptedException
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1934)
        	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        	at org.apache.bookkeeper.bookie.Bookie.run(Bookie.java:341)
        

        and the assert statement that fails:

        Testcase: testNoLedger took 0.132 sec
        	FAILED
        expected:<1> but was:<2>
        junit.framework.AssertionFailedError: expected:<1> but was:<2>
        	at org.apache.bookkeeper.test.BookieClientTest.testNoLedger(BookieClientTest.java:204)
        

        It seems to me that the problem is that the bookie is not processing the FileNotFoundException properly. It should send a ENOLEDGER value back.

        Show
        Flavio Junqueira added a comment - Here is the log output for the failing test: 2009-12-02 09:46:22,952 - INFO - [main:BookieClientTest@198] - Starting no ledger test 2009-12-02 09:46:22,956 - INFO - [Thread-4:BookieClientTest$1@75] - Capacity 16, 0 2009-12-02 09:46:22,957 - INFO - [Thread-4:BookieClientTest$1@83] - Received 0 2009-12-02 09:46:23,001 - ERROR - [SyncThread:Bookie$LastLogMark@295] - Problems writing to /var/folders/iq/iqmyL+XiGjqE2AjRkKTe-E+++TI/-Tmp-/bookie2550661167481866393test/lastMark java.io.FileNotFoundException: /var/folders/iq/iqmyL+XiGjqE2AjRkKTe-E+++TI/-Tmp-/bookie2550661167481866393test/lastMark (No such file or directory) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:179) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at org.apache.bookkeeper.bookie.Bookie$LastLogMark.rollLog(Bookie.java:290) at org.apache.bookkeeper.bookie.Bookie$SyncThread.run(Bookie.java:116) 2009-12-02 09:46:23,002 - FATAL - [Thread-3:Bookie@375] - Bookie thread exiting java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1934) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.bookkeeper.bookie.Bookie.run(Bookie.java:341) and the assert statement that fails: Testcase: testNoLedger took 0.132 sec FAILED expected:<1> but was:<2> junit.framework.AssertionFailedError: expected:<1> but was:<2> at org.apache.bookkeeper.test.BookieClientTest.testNoLedger(BookieClientTest.java:204) It seems to me that the problem is that the bookie is not processing the FileNotFoundException properly. It should send a ENOLEDGER value back.
        Hide
        Benjamin Reed added a comment -

        fixed the check for no lodger in bookie.java

        Show
        Benjamin Reed added a comment - fixed the check for no lodger in bookie.java
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12427942/ZOOKEEPER-486.patch
        against trunk revision 889848.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/87/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/87/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/87/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427942/ZOOKEEPER-486.patch against trunk revision 889848. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/87/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/87/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/87/console This message is automatically generated.
        Hide
        Benjamin Reed added a comment -

        added comments

        Show
        Benjamin Reed added a comment - added comments
        Hide
        Flavio Junqueira added a comment -

        Ben, this patch is broken. It is missing a few classes, like LedgerCache and LedgerEntryPage.

        Show
        Flavio Junqueira added a comment - Ben, this patch is broken. It is missing a few classes, like LedgerCache and LedgerEntryPage.
        Hide
        Benjamin Reed added a comment -

        added the missing files.

        Show
        Benjamin Reed added a comment - added the missing files.
        Hide
        Benjamin Reed added a comment -

        the patch was rooted in bookkeeper rather than the top.

        Show
        Benjamin Reed added a comment - the patch was rooted in bookkeeper rather than the top.
        Hide
        Flavio Junqueira added a comment -

        This issue has been fixed as part of the patch of ZOOKEEPER-507.

        Show
        Flavio Junqueira added a comment - This issue has been fixed as part of the patch of ZOOKEEPER-507 .

          People

          • Assignee:
            Benjamin Reed
            Reporter:
            Flavio Junqueira
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development