Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-895

Allow hflush/sync to occur in parallel with new writes to the file

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      In the current trunk, the HDFS client methods writeChunk() and hflush./sync are syncronized. This means that if a hflush/sync is in progress, an applicationn cannot write data to the HDFS client buffer. This reduces the write throughput of the transaction log in HBase.

      The hflush/sync should allow new writes to happen to the HDFS client even when a hflush/sync is in progress. It can record the seqno of the message for which it should receice the ack, indicate to the DataStream thread to star flushing those messages, exit the synchronized section and just wai for that ack to arrive.

      Attachments

        1. hdfs-895-review.txt
          20 kB
          Todd Lipcon
        2. hdfs-895-ontopof-1497.txt
          23 kB
          Todd Lipcon
        3. hdfs-895-branch-20-append.txt
          17 kB
          Todd Lipcon
        4. hdfs-895-20.txt
          16 kB
          Todd Lipcon
        5. hdfs-895-0.20-append.txt
          16 kB
          Todd Lipcon
        6. hdfs-895-0.20-append.txt
          19 kB
          Todd Lipcon
        7. hdfs-895.txt
          17 kB
          Todd Lipcon
        8. hdfs-895.txt
          21 kB
          Todd Lipcon
        9. hdfs-895.txt
          22 kB
          Todd Lipcon
        10. hdfs-895.txt
          22 kB
          Todd Lipcon
        11. hdfs-895.txt
          21 kB
          Todd Lipcon
        12. hdfs-895.txt
          22 kB
          Todd Lipcon
        13. HDFS-895.20-security.1.patch
          18 kB
          Jitendra Nath Pandey
        14. ASF.LICENSE.NOT.GRANTED--hdfs-895-trunk.txt
          19 kB
          Todd Lipcon
        15. ASF.LICENSE.NOT.GRANTED--hdfs-895.txt
          15 kB
          Todd Lipcon
        16. 895-delta-for-review.txt
          7 kB
          Todd Lipcon

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-2467 Concurrent flushers in HLog sync using HDFS-895

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-1497 Write pipeline sequence numbers should be sequential with no skips or duplicates

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              tlipcon Todd Lipcon
              dhruba Dhruba Borthakur
              Votes:
              1 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: