Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1906

Ability to disable WAL for put operation in HBaseSink

        Details

        • Type: Improvement
        • Status: Resolved
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: 1.3.1
        • Fix Version/s: 1.4.0
        • Component/s: Sinks+Sources
        • Labels:
          None

          Description

          HBase supports setWriteToWAL(boolean) in Put to write edits to WAL or not. User can specify the disable option in HBase sink configurations.

          1. Text File
            FLUME-1906.patch
            5 kB
            Hari Shreedharan
          2. Text File
            FLUME-1906-1.patch
            5 kB
            Hari Shreedharan

            Issue Links

            links to

            Web Link Review

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              hshreedharan Hari Shreedharan added a comment -

              Doing this would void Flume's delivery guarantee no? (not exactly Flume's guarantee, but still the data would not be persisted to disk).

              Show
              hshreedharan Hari Shreedharan added a comment - Doing this would void Flume's delivery guarantee no? (not exactly Flume's guarantee, but still the data would not be persisted to disk).
              Hide
              Permalink
              mubarak.seyed Mubarak Seyed added a comment -

              Doing this would void Flume's delivery guarantee no? (not exactly Flume's guarantee, but still the data would not be persisted to disk).

              If Region-server crashes then whatever data in memstore will be lost (provided region flush size is tuned appropriately). This option helps hbase ops to dynamically enable/diable WAL to save disk space in HDFS (i think flume agent needs to be restarted to see the updated hbase sink config?).

              Show
              mubarak.seyed Mubarak Seyed added a comment - Doing this would void Flume's delivery guarantee no? (not exactly Flume's guarantee, but still the data would not be persisted to disk). If Region-server crashes then whatever data in memstore will be lost (provided region flush size is tuned appropriately). This option helps hbase ops to dynamically enable/diable WAL to save disk space in HDFS (i think flume agent needs to be restarted to see the updated hbase sink config?).
              Hide
              Permalink
              hshreedharan Hari Shreedharan added a comment -

              The flume agent does not need to restart. If the config file is updated, flume will pick it up automatically and reload all components. I am just a bit concerned about completing the transaction on the sink side without the data being persisted. This could potentially cause debugging issues.

              Show
              hshreedharan Hari Shreedharan added a comment - The flume agent does not need to restart. If the config file is updated, flume will pick it up automatically and reload all components. I am just a bit concerned about completing the transaction on the sink side without the data being persisted. This could potentially cause debugging issues.
              Hide
              Permalink
              mubarakseyed Mubarak Seyed added a comment -

              I am just a bit concerned about completing the transaction on the sink side without the data being persisted. This could potentially cause debugging issues.

              I think it is a trade-off by giving up durability (for yet-to-flush in-memory data) to save disk space in HDFS. There are some instances where ops runs hbase clusters with WAL turned off to save 3 copies of WAL data in HDFS. if config changes for WAL then we can add a INFO in log for easy debugging.

              Show
              mubarakseyed Mubarak Seyed added a comment - I am just a bit concerned about completing the transaction on the sink side without the data being persisted. This could potentially cause debugging issues. I think it is a trade-off by giving up durability (for yet-to-flush in-memory data) to save disk space in HDFS. There are some instances where ops runs hbase clusters with WAL turned off to save 3 copies of WAL data in HDFS. if config changes for WAL then we can add a INFO in log for easy debugging.
              Hide
              Permalink
              mubarakseyed Mubarak Seyed added a comment -

              Thanks for the patch Hari! Pushed to trunk & flume-1.4 branch.

              Show
              mubarakseyed Mubarak Seyed added a comment - Thanks for the patch Hari! Pushed to trunk & flume-1.4 branch.
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in flume-trunk #362 (See https://builds.apache.org/job/flume-trunk/362/)
              FLUME-1906 Ability to disable WAL for put operation in HBaseSink (Revision 510f63ba39592e7912a85c35effb8be52699057a)

              Result = SUCCESS
              mubarak : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=510f63ba39592e7912a85c35effb8be52699057a
              Files :

              • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java
              • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java
              • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java
              Show
              hudson Hudson added a comment - Integrated in flume-trunk #362 (See https://builds.apache.org/job/flume-trunk/362/ ) FLUME-1906 Ability to disable WAL for put operation in HBaseSink (Revision 510f63ba39592e7912a85c35effb8be52699057a) Result = SUCCESS mubarak : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=510f63ba39592e7912a85c35effb8be52699057a Files : flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java
              Hide
              Permalink
              hshreedharan Hari Shreedharan added a comment -

              Adding the review link for posterity.

              Show
              hshreedharan Hari Shreedharan added a comment - Adding the review link for posterity.

                People

                • Assignee:
                  hshreedharan Hari Shreedharan
                  Reporter:
                  mubarakseyed Mubarak Seyed
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development