Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.4.3, 3.4.4, 3.5.0
    • Fix Version/s: 3.5.0
    • Component/s: server
    • Hadoop Flags:
      Reviewed

      Description

      CommitProcessor has a single thread that both pulls requests off its queues and runs all downstream processors. This is noticeably inefficient for read-intensive workloads, which could be run concurrently. The trick is handling write transactions. I propose multi-threading this code according to the following two constraints

      • each session must see its requests responded to in order
      • all committed transactions must be handled in zxid order, across all sessions

      I believe these cover the only constraints we need to honor. In particular, I believe we can relax the following:

      • it does not matter if the read request in one session happens before or after the write request in another session

      With these constraints, I propose the following threads

      • 1 primary queue servicing/work dispatching thread
      • 0-N assignable worker threads, where a given session is always assigned to the same worker thread

      By assigning sessions always to the same worker thread (using a simple sessionId mod number of worker threads), we guarantee the first constraint-- requests we push onto the thread queue are processed in order. The way we guarantee the second constraint is we only allow a single commit transaction to be in flight at a time--the queue servicing thread blocks while a commit transaction is in flight, and when the transaction completes it clears the flag.

      On a 32 core machine running Linux 2.6.38, achieved best performance with 32 worker threads for a 56% +/- 5% improvement in throughput (this improvement was measured on top of that for ZOOKEEPER-1504, not in isolation).

      New classes introduced in this patch are:

      WorkerService (also in ZOOKEEPER-1504): ExecutorService wrapper that makes worker threads daemon threads and names then in an easily debuggable manner. Supports assignable threads (as used here) and non-assignable threads (as used by NIOServerCnxnFactory).

        Attachments

        1. ZOOKEEPER-1505.patch
          31 kB
          Jay Shrauner
        2. ZOOKEEPER-1505.patch
          48 kB
          Jay Shrauner
        3. ZOOKEEPER-1505.patch
          48 kB
          Jay Shrauner
        4. ZOOKEEPER-1505.patch
          49 kB
          Patrick Hunt

          Activity

            People

            • Assignee:
              shrauner Jay Shrauner
              Reporter:
              shrauner Jay Shrauner
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: