HBase
  1. HBase
  2. HBASE-3150

Allow some column to not write WALs

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Coprocessors
    • Labels:
      None

      Description

      We have this unique requirement where some column families hold data that is indexed from other existing column families. The index data is very large, and we end up writing these inserts into the WAL and then into the store files. In addition to taking more iops, this also slows down splitting files for recovery, etc.

      Creating this task to have an option to suppress WAL logging on a per CF basis.

        Activity

        Karthik Ranganathan created issue -
        Hide
        Mingjie Lai added a comment -

        A WAL coprocessor (HBASE-3257) can help to solve the problem.

        By a WAL cp implementation, one or multiple specified columns can be removed from the passed WALEdit before appending to WAL, so that they will be ignored at WAL. However they will still be written to store files.

        Another option is to utilize existing WALObserver which provides a hook to insert user code before HLog.doWrite(). But currently it doesn't support to modify the passed WALEdit. I think Coprocessor is a better way to do it.

        Show
        Mingjie Lai added a comment - A WAL coprocessor ( HBASE-3257 ) can help to solve the problem. By a WAL cp implementation, one or multiple specified columns can be removed from the passed WALEdit before appending to WAL, so that they will be ignored at WAL. However they will still be written to store files. Another option is to utilize existing WALObserver which provides a hook to insert user code before HLog.doWrite(). But currently it doesn't support to modify the passed WALEdit. I think Coprocessor is a better way to do it.
        Hide
        Karthik Ranganathan added a comment -

        Yes, a co-processors based implementation would totally work.

        Show
        Karthik Ranganathan added a comment - Yes, a co-processors based implementation would totally work.
        Hide
        Kannan Muthukkaruppan added a comment -

        One workaround would be to use:

        Put.writeToWAL(false).

        Caveat: This wouldn't work for the case when you have a mix of column families in the Put, some of which you want to write to WAL and others you don't.

        Show
        Kannan Muthukkaruppan added a comment - One workaround would be to use: Put.writeToWAL(false). Caveat: This wouldn't work for the case when you have a mix of column families in the Put, some of which you want to write to WAL and others you don't.
        Todd Lipcon made changes -
        Field Original Value New Value
        Component/s coprocessors [ 12314191 ]
        Hide
        Mingjie Lai added a comment -

        Karthik.
        Please take a look at the patch at reviewboard for HBase-3257 – https://review.cloudera.org/r/1515/. There is a SampleRegionWALObserver which can add, remove, modify KeyValues to a WALEdit before written to WAL.

        Show
        Mingjie Lai added a comment - Karthik. Please take a look at the patch at reviewboard for HBase-3257 – https://review.cloudera.org/r/1515/ . There is a SampleRegionWALObserver which can add, remove, modify KeyValues to a WALEdit before written to WAL.
        Hide
        Andrew Purtell added a comment -

        Stale issue. Reopen if still relevant (if there's new activity)

        Show
        Andrew Purtell added a comment - Stale issue. Reopen if still relevant (if there's new activity)
        Andrew Purtell made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        1322d 1h 8m 1 Andrew Purtell 08/Jun/14 23:04

          People

          • Assignee:
            Unassigned
            Reporter:
            Karthik Ranganathan
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development