Details

    • Type: New Feature New Feature
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I've been working on a generic JDBC sink. It needs a bit more testing, but I think it's ready for review and feedback. I have not yet updated the Flume documentation, but I can / will if people are happy with this.

      Since the config file is how you’d interact with it, here’s a working example from my source tree:

      a.sinks.k.type=jdbc
      a.sinks.k.channel=c
      a.sinks.k.driver=com.mysql.jdbc.Driver
      a.sinks.k.url=jdbc:mysql://localhost:8889/flume
      a.sinks.k.user=username
      a.sinks.k.password=password
      a.sinks.k.batchSize=100
      a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, ${header.timestamp:long})
      

      The interesting part is the SQL statement. You can put anything you want in there - it will get converted to a prepared statement on execution. The Ant-ish tokens get parsed and replaced with parameters at startup.

      The tokens are three part. For example, in:

      ${body:string(UTF-8)}
      

      The first is a place in the event to get the value from (“body”, “header.foo”, or “custom”). The second part ("string") is a type identifier that converts into an appropriate JDBC parameter. The third part (“UTF-8") is a configuration string for that type, if needed. As for types, so far I’ve defined:

      body: string (with optional charset encoding), bytearray
      header: string, long, int, float, double, date (with mandatory date format and optional timezone)

      Additionally, if none of those make you happy you can define you own parameter converters:

      ${custom:com.company.foo.MyConverter(optionaltextconfig)}
      
      1. FLUME-2256.diff
        79 kB
        Jeremy Karlson

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Jeremy Karlson
              Reporter:
              Jeremy Karlson
            • Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development