Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1945

HBase Serializer allow key from regular expression group

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.1
    • Fix Version/s: 1.4.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      Allow users to chose a regular expression group as hbase row key. To use this feature users need to:

      1. Specify the row key index.
      2. Specify ROW_KEY as the column name for the corresponding index

      For example:
      a1.sinks.k1.serializer.rowKeyIndex = 1
      a1.sinks.k1.serializer.colNames = host,ROW_KEY,timestamp
      Show
      Allow users to chose a regular expression group as hbase row key. To use this feature users need to: 1. Specify the row key index. 2. Specify ROW_KEY as the column name for the corresponding index For example: a1.sinks.k1.serializer.rowKeyIndex = 1 a1.sinks.k1.serializer.colNames = host,ROW_KEY,timestamp

      Description

      Back when we created the the HBase serializer interface we decided that most people would have to implement their own since they knew their data and generically mapping an event to columns and keys was likely going to be difficult. I have observed that something like the following: https://github.com/brockn/flume-hbase-serializer

      would be useful to users. Basically it's the regex serializer where a single value from the regex match can be used as the key. It might be possible to modify the current regex serializer to behave in this way so as to limit the number of serializers.

      1. FLUME-1945.patch
        8 kB
        Sravya Tirukkovalur
      2. FLUME-1945.patch
        5 kB
        Sravya Tirukkovalur

        Issue Links

          Activity

          Hide
          sravya Sravya Tirukkovalur added a comment -

          I would like to pick this Jira.

          Thanks!

          Show
          sravya Sravya Tirukkovalur added a comment - I would like to pick this Jira. Thanks!
          Hide
          sravya Sravya Tirukkovalur added a comment -

          Updated Patch

          Show
          sravya Sravya Tirukkovalur added a comment - Updated Patch
          Hide
          brocknoland Brock Noland added a comment -

          Committed to trunk and 1.4. Thanks Sravya!!

          Show
          brocknoland Brock Noland added a comment - Committed to trunk and 1.4. Thanks Sravya!!
          Hide
          hudson Hudson added a comment -

          Integrated in flume-trunk #420 (See https://builds.apache.org/job/flume-trunk/420/)
          FLUME-1945: HBase Serializer allow key from regular expression group (Revision e442c29a646723a75d67fe4dd189cc304dfd4af4)

          Result = FAILURE
          brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=e442c29a646723a75d67fe4dd189cc304dfd4af4
          Files :

          • flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestRegexHbaseEventSerializer.java
          • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/RegexHbaseEventSerializer.java
          Show
          hudson Hudson added a comment - Integrated in flume-trunk #420 (See https://builds.apache.org/job/flume-trunk/420/ ) FLUME-1945 : HBase Serializer allow key from regular expression group (Revision e442c29a646723a75d67fe4dd189cc304dfd4af4) Result = FAILURE brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=e442c29a646723a75d67fe4dd189cc304dfd4af4 Files : flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestRegexHbaseEventSerializer.java flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/RegexHbaseEventSerializer.java
          Hide
          sravya Sravya Tirukkovalur added a comment -

          Thanks Brock, for the commit!

          Show
          sravya Sravya Tirukkovalur added a comment - Thanks Brock, for the commit!

            People

            • Assignee:
              sravya Sravya Tirukkovalur
              Reporter:
              brocknoland Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development