Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-7329

remove flush-related records from WAL and make locking more granular

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.95.2
    • Fix Version/s: 0.95.0
    • Component/s: wal
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      memstore flush WAL record is removed.
      Previously, there existed cacheFlushLock which was used for 4 distinct purposes:
      1) Make sure the WAL record for memstore flush ends up in the same WAL that was active when the flush started. In other words, log rolling is prevented during memstore flush.
      2) Make sure only one log rolling operation happens at a time.
      3) Control access to "lastSeqWritten" map for any operation that is not covered by SynchronizedMap.
      4) Make sure that close() waits for all the flushes and rolls, by virtue of doing work under this lock and setting .closed = true for potential future callers.
      Nothing prevented startCacheFlush from proceeding after close is done, since it doesn't perform the check of closed.
      cacheFlushLock was replaced by more granular/appropriate locking:
      1) No longer needed/relevant, in fact there's HBASE-7011 to get rid of this too.
      2) rollWriter is made "synchronized".
      3) Control of the map is under its own small-scope lock.
      4) Added a simple barrier class to be able to block close on outstanding operations, and drain existing ones.
      Show
      memstore flush WAL record is removed. Previously, there existed cacheFlushLock which was used for 4 distinct purposes: 1) Make sure the WAL record for memstore flush ends up in the same WAL that was active when the flush started. In other words, log rolling is prevented during memstore flush. 2) Make sure only one log rolling operation happens at a time. 3) Control access to "lastSeqWritten" map for any operation that is not covered by SynchronizedMap. 4) Make sure that close() waits for all the flushes and rolls, by virtue of doing work under this lock and setting .closed = true for potential future callers. Nothing prevented startCacheFlush from proceeding after close is done, since it doesn't perform the check of closed. cacheFlushLock was replaced by more granular/appropriate locking: 1) No longer needed/relevant, in fact there's HBASE-7011 to get rid of this too. 2) rollWriter is made "synchronized". 3) Control of the map is under its own small-scope lock. 4) Added a simple barrier class to be able to block close on outstanding operations, and drain existing ones.

      Description

      Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush records in WAL are not useful. If so, they should be removed.

        Attachments

        1. 7329-findbugs.diff
          24 kB
          Ted Yu
        2. 7329-v7.txt
          54 kB
          Ted Yu
        3. HBASE-7329-v0.patch
          36 kB
          Sergey Shelukhin
        4. HBASE-7329-v0.patch
          36 kB
          Sergey Shelukhin
        5. HBASE-7329-v0-tmp.patch
          38 kB
          Sergey Shelukhin
        6. HBASE-7329-v1.patch
          45 kB
          Sergey Shelukhin
        7. HBASE-7329-v1.patch
          45 kB
          Sergey Shelukhin
        8. HBASE-7329-v2.patch
          49 kB
          Sergey Shelukhin
        9. HBASE-7329-v3.patch
          50 kB
          Sergey Shelukhin
        10. HBASE-7329-v4.patch
          55 kB
          Sergey Shelukhin
        11. HBASE-7329-v5.patch
          55 kB
          Sergey Shelukhin
        12. HBASE-7329-v6.patch
          55 kB
          Sergey Shelukhin
        13. HBASE-7329-v6.patch
          55 kB
          Sergey Shelukhin

          Issue Links

            Activity

              People

              • Assignee:
                sershe Sergey Shelukhin
                Reporter:
                sershe Sergey Shelukhin
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: