Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2330

Preallocate transaction log to improve namenode transaction logging performance

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.2
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In the current implementation, the transaction log is opened in "append" mode and every new transaction is written to the end of the log. This means that new blocks get allocated to the edits file frequently.

      It is worth measuring the performance improvement when big chunks of the transaction log are allocated up front. Adding new transactions do not cause frequent block allocations for the edits log.

        Attachments

        1. preallocateTransactionLog4.patch
          4 kB
          Hairong Kuang
        2. preallocateTransactionLog3.patch
          6 kB
          Dhruba Borthakur
        3. preallocateTransactionLog2.patch
          6 kB
          Dhruba Borthakur
        4. preallocateTransactionLog.patch
          6 kB
          Dhruba Borthakur
        5. preallocateTransactionLog.patch
          6 kB
          Dhruba Borthakur

          Issue Links

            Activity

              People

              • Assignee:
                dhruba Dhruba Borthakur
                Reporter:
                dhruba Dhruba Borthakur
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: