Flume
  1. Flume
  2. FLUME-1757

Improve configuration of hbase serializers

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: v1.3.0
    • Fix Version/s: v1.4.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Adding default values for both hbase and aysnc hbase simple serializer's properties. Default values are "iCol" and "pCol" for payload and increment columns respectively.

      Description

      For example HBase SimpleHbaseEventSerializer should throw an exception if neither payloadColumn or IncrementColumn are specified. If neither are specified the serializer writes out nothing which is very confusing to users.

      1. 1757.patch
        8 kB
        Sravya Tirukkovalur

        Issue Links

          Activity

          Hide
          Sravya Tirukkovalur added a comment -

          I can pick up this Jira. Brock, so both payloadColumn and IncrementColumn are required when using SimpleHbaseEventSerializer, is that correct?

          Show
          Sravya Tirukkovalur added a comment - I can pick up this Jira. Brock, so both payloadColumn and IncrementColumn are required when using SimpleHbaseEventSerializer, is that correct?
          Hide
          Hari Shreedharan added a comment -

          Sravya Tirukkovalur, not necessarily both - but we should throw if we don't get at least one of the two. Also, we do pick up the default serializer - so we need to specify serializer.property without specifying the serializer itself - which seems awkward. We should just pass in the entire context into the serializer (to make sure that we are backward compatible, we will need to strip off the "serializer." part).

          Show
          Hari Shreedharan added a comment - Sravya Tirukkovalur , not necessarily both - but we should throw if we don't get at least one of the two. Also, we do pick up the default serializer - so we need to specify serializer.property without specifying the serializer itself - which seems awkward. We should just pass in the entire context into the serializer (to make sure that we are backward compatible, we will need to strip off the "serializer." part).
          Hide
          Sravya Tirukkovalur added a comment -

          Instead of providing a default serializer which has required properties, wouldnt it be safe to have default values for payloadColumn and IncrementColumn and make them optional? And make a note in the documentation about what are the default values for payloadColumn and IncrementColumn?

          Show
          Sravya Tirukkovalur added a comment - Instead of providing a default serializer which has required properties, wouldnt it be safe to have default values for payloadColumn and IncrementColumn and make them optional? And make a note in the documentation about what are the default values for payloadColumn and IncrementColumn?
          Hide
          Hari Shreedharan added a comment -

          +1 - makes sense.

          Show
          Hari Shreedharan added a comment - +1 - makes sense.
          Hide
          Hari Shreedharan added a comment -

          +1 (this time, for the patch).

          Show
          Hari Shreedharan added a comment - +1 (this time, for the patch).
          Hide
          Hari Shreedharan added a comment -

          Patch committed, rev: 669e5d3. Thanks Sravya!

          Show
          Hari Shreedharan added a comment - Patch committed, rev: 669e5d3. Thanks Sravya!
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #414 (See https://builds.apache.org/job/flume-trunk/414/)
          FLUME-1757. Improve configuration of hbase serializers. (Revision 669e5d327720021391245949d6a947b4e963b728)

          Result = SUCCESS
          hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=669e5d327720021391245949d6a947b4e963b728
          Files :

          • flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java
          • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/SimpleAsyncHbaseEventSerializer.java
          • flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestHBaseSink.java
          • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/SimpleHbaseEventSerializer.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          Show
          Hudson added a comment - Integrated in flume-trunk #414 (See https://builds.apache.org/job/flume-trunk/414/ ) FLUME-1757 . Improve configuration of hbase serializers. (Revision 669e5d327720021391245949d6a947b4e963b728) Result = SUCCESS hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=669e5d327720021391245949d6a947b4e963b728 Files : flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/SimpleAsyncHbaseEventSerializer.java flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestHBaseSink.java flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/SimpleHbaseEventSerializer.java flume-ng-doc/sphinx/FlumeUserGuide.rst
          Hide
          Alexander Alten-Lorenz added a comment -

          Since the patch is in trunk we should close the review request:
          https://reviews.apache.org/r/10989/

          Show
          Alexander Alten-Lorenz added a comment - Since the patch is in trunk we should close the review request: https://reviews.apache.org/r/10989/

            People

            • Assignee:
              Sravya Tirukkovalur
              Reporter:
              Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development