HBase
  1. HBase
  2. HBASE-1416

Pool of commit loggers in each HRegionServer

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Later
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      HBASE-1394 discusses pools of loggers as means of our being able to dump out the logs faster; commit log is log pole in a write transaction. This issue is about implementing the pool of writers.

        Issue Links

          Activity

          Hide
          Andrew Purtell added a comment -

          Up on hbase-dev@ Ryan writes:

          we need to make hlog flush faster, it currently does only 700 ops/sec
          when we flush every entry.

          it'd be nice if we could do something clever, such as:

          • use multiple logs
          • detect multiple waiting clients and better batch their commits
          • group commits for bulk import

          This issue addresses the first point.

          While considering this, dynamically size the pool according to a concurrency measure. Spin up new writers on demand until some configurable upper bound. A simple strategy to try first might be 2 * ceil(log(load)), smoothed. Terminate excess writers at roll time to hold down unnecessary HDFS resource use.

          In HLog.doWrite we write each HLogKey and KeyValue to the log, which is a SequenceFile. Use hfile instead? Can HFile do I/O batching? Otherwise I think to group commits we'd need to introduce a new writable which bundles edits together.

          Moving into 0.21.

          Show
          Andrew Purtell added a comment - Up on hbase-dev@ Ryan writes: we need to make hlog flush faster, it currently does only 700 ops/sec when we flush every entry. it'd be nice if we could do something clever, such as: use multiple logs detect multiple waiting clients and better batch their commits group commits for bulk import This issue addresses the first point. While considering this, dynamically size the pool according to a concurrency measure. Spin up new writers on demand until some configurable upper bound. A simple strategy to try first might be 2 * ceil(log(load)), smoothed. Terminate excess writers at roll time to hold down unnecessary HDFS resource use. In HLog.doWrite we write each HLogKey and KeyValue to the log, which is a SequenceFile. Use hfile instead? Can HFile do I/O batching? Otherwise I think to group commits we'd need to introduce a new writable which bundles edits together. Moving into 0.21.
          Hide
          Andrew Purtell added a comment -

          Split out writes into a commit thread per WAL after HBASE-1939.

          Show
          Andrew Purtell added a comment - Split out writes into a commit thread per WAL after HBASE-1939 .
          Hide
          Andrew Purtell added a comment -

          Do we still want to do this? Seems some of Ryan's ideas have been implemented as HBASE-1939 already.

          I was playing around with this the other day after committing HBASE-2059. On that note, implementing an HLog.Writer that stripes edits over a pool of logs under a directory instead of a single file works ok on read and write paths but split and roll code assumes the log is a single file so barf when they access a dir instead; the HLog.Writer interface would need to be widened with split and roll methods to make that work cleanly. Anyway, sync is still a log pole and with striped writes over a pool of HLogs there are more dirty blocks to sync. Unless the current edit load is high we end up dirtying small bits of a bunch of blocks where instead we should just write them all into one as they'll all fit in there.

          Show
          Andrew Purtell added a comment - Do we still want to do this? Seems some of Ryan's ideas have been implemented as HBASE-1939 already. I was playing around with this the other day after committing HBASE-2059 . On that note, implementing an HLog.Writer that stripes edits over a pool of logs under a directory instead of a single file works ok on read and write paths but split and roll code assumes the log is a single file so barf when they access a dir instead; the HLog.Writer interface would need to be widened with split and roll methods to make that work cleanly. Anyway, sync is still a log pole and with striped writes over a pool of HLogs there are more dirty blocks to sync. Unless the current edit load is high we end up dirtying small bits of a bunch of blocks where instead we should just write them all into one as they'll all fit in there.
          Hide
          stack added a comment -

          Below are Ryan's comments from the list.

          "With deferred log flush we can choose between high durability and speed on a
          per table basis. I think we should shelf this for now, since with write
          buffering in the client and the adjusted sync calls, we are very fast at
          bulk data import."

          J-D says write speeds are up with hbase-1939.

          I'm good w/ punting out to 0.22 or beyond. Let me move it.

          Show
          stack added a comment - Below are Ryan's comments from the list. "With deferred log flush we can choose between high durability and speed on a per table basis. I think we should shelf this for now, since with write buffering in the client and the adjusted sync calls, we are very fast at bulk data import." J-D says write speeds are up with hbase-1939. I'm good w/ punting out to 0.22 or beyond. Let me move it.
          Hide
          stack added a comment -

          Moved it out of 0.21 and made it trivial priority.

          Show
          stack added a comment - Moved it out of 0.21 and made it trivial priority.
          Hide
          Andrew Purtell added a comment -

          Resolving as Later. I believe everyone agrees this issue is something to defer, probably indefinitely.

          Show
          Andrew Purtell added a comment - Resolving as Later. I believe everyone agrees this issue is something to defer, probably indefinitely.
          Hide
          Jean-Daniel Cryans added a comment -

          I agree, since we always write on local node first then having multiple writers won't help. Also the 2 latter points were implemented so the appending/hflushing situation is actually pretty good.

          Show
          Jean-Daniel Cryans added a comment - I agree, since we always write on local node first then having multiple writers won't help. Also the 2 latter points were implemented so the appending/hflushing situation is actually pretty good.

            People

            • Assignee:
              Unassigned
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development