Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1942

Increase the concurrency of transaction logging to edits log

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.15.0
    • Component/s: None
    • Labels:
      None

      Description

      For some typical workloads, the throughput of the namenode is bottlenecked by the rate of transactions that are being logged into tghe edits log. In the current code, a batching scheme implies that all transactions do not have to incur a sync of the edits log to disk. However, the existing batch-ing scheme can be improved.

      One option is to keep two buffers associated with edits file. Threads write to the primary buffer while holding the FSNamesystem lock. Then the thread release the FSNamesystem lock, acquires a new lock called the syncLock, swaps buffers, and flushes the old buffer to the persistent store. Since the buffers are swapped, new transactions continue to get logged into the new buffer. (Of course, the new transactions cannot complete before this new buffer is sync-ed).

      This approach does a better job of batching syncs to disk, thus improving performance.

        Attachments

        1. transactionLogSync.patch
          24 kB
          Dhruba Borthakur
        2. transactionLogSync2.patch
          29 kB
          Dhruba Borthakur
        3. transactionLogSync3.patch
          25 kB
          Dhruba Borthakur
        4. transactionLogSync4.patch
          25 kB
          Dhruba Borthakur
        5. transactionLogSync5.patch
          26 kB
          Dhruba Borthakur
        6. transactionLogSync6.patch
          26 kB
          Dhruba Borthakur
        7. transactionLogSync8.patch
          26 kB
          Dhruba Borthakur
        8. transactionLogSync9.patch
          26 kB
          Dhruba Borthakur

          Issue Links

            Activity

              People

              • Assignee:
                dhruba Dhruba Borthakur
                Reporter:
                dhruba Dhruba Borthakur
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: