Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2330

Preallocate transaction log to improve namenode transaction logging performance

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.17.2
    • 0.19.0
    • None
    • None
    • Reviewed

    Description

      In the current implementation, the transaction log is opened in "append" mode and every new transaction is written to the end of the log. This means that new blocks get allocated to the edits file frequently.

      It is worth measuring the performance improvement when big chunks of the transaction log are allocated up front. Adding new transactions do not cause frequent block allocations for the edits log.

      Attachments

        1. preallocateTransactionLog.patch
          6 kB
          Dhruba Borthakur
        2. preallocateTransactionLog.patch
          6 kB
          Dhruba Borthakur
        3. preallocateTransactionLog2.patch
          6 kB
          Dhruba Borthakur
        4. preallocateTransactionLog3.patch
          6 kB
          Dhruba Borthakur
        5. preallocateTransactionLog4.patch
          4 kB
          Hairong Kuang

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-3510 Improve FSEditLog pre-allocation

          • Major - Major loss of function.
          • Closed

          Activity

            People

              dhruba Dhruba Borthakur
              dhruba Dhruba Borthakur
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: