Details

    • Type: New Feature New Feature
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I've been working on a generic JDBC sink. It needs a bit more testing, but I think it's ready for review and feedback. I have not yet updated the Flume documentation, but I can / will if people are happy with this.

      Since the config file is how you’d interact with it, here’s a working example from my source tree:

      a.sinks.k.type=jdbc
      a.sinks.k.channel=c
      a.sinks.k.driver=com.mysql.jdbc.Driver
      a.sinks.k.url=jdbc:mysql://localhost:8889/flume
      a.sinks.k.user=username
      a.sinks.k.password=password
      a.sinks.k.batchSize=100
      a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string}, ${header.timestamp:long})
      

      The interesting part is the SQL statement. You can put anything you want in there - it will get converted to a prepared statement on execution. The Ant-ish tokens get parsed and replaced with parameters at startup.

      The tokens are three part. For example, in:

      ${body:string(UTF-8)}
      

      The first is a place in the event to get the value from (“body”, “header.foo”, or “custom”). The second part ("string") is a type identifier that converts into an appropriate JDBC parameter. The third part (“UTF-8") is a configuration string for that type, if needed. As for types, so far I’ve defined:

      body: string (with optional charset encoding), bytearray
      header: string, long, int, float, double, date (with mandatory date format and optional timezone)

      Additionally, if none of those make you happy you can define you own parameter converters:

      ${custom:com.company.foo.MyConverter(optionaltextconfig)}
      
      1. FLUME-2256.diff
        79 kB
        Jeremy Karlson

        Issue Links

          Activity

          Hide
          Santiago M. Mola added a comment - - edited

          I haven't got time recently to work on this, but there is a work in progress to prepare the JDBC sink for submission. It should be already functional but it still lacks a lot of testing. Anyone interested, here's the link to the exact branch:
          https://github.com/Stratio/flume/tree/features/jdbc-sink/flume-ng-sinks/flume-ng-jdbc-sink

          Show
          Santiago M. Mola added a comment - - edited I haven't got time recently to work on this, but there is a work in progress to prepare the JDBC sink for submission. It should be already functional but it still lacks a lot of testing. Anyone interested, here's the link to the exact branch: https://github.com/Stratio/flume/tree/features/jdbc-sink/flume-ng-sinks/flume-ng-jdbc-sink
          Hide
          Santiago M. Mola added a comment -

          We have developed a JDBC Sink inspired by Jeremy's and based on jOOQ. It can either use a custom SQL query to insert data or automatically map the input to an existing table. You can find it at: https://github.com/Stratio/stratio-ingestion/tree/develop/stratio-sinks/stratio-jdbc-sink

          Show
          Santiago M. Mola added a comment - We have developed a JDBC Sink inspired by Jeremy's and based on jOOQ. It can either use a custom SQL query to insert data or automatically map the input to an existing table. You can find it at: https://github.com/Stratio/stratio-ingestion/tree/develop/stratio-sinks/stratio-jdbc-sink
          Hide
          Viktor Trako added a comment -

          Where is this at the moment - has anyone tested this patch further?

          Currently using it in a staging environment and its working beautifully for writing events to MySql server. The dependencies required for this patch are working in harmony with my current agent requirements.

          Dissapointed this didnt make its way through to 1.5 release.

          Viktor

          Show
          Viktor Trako added a comment - Where is this at the moment - has anyone tested this patch further? Currently using it in a staging environment and its working beautifully for writing events to MySql server. The dependencies required for this patch are working in harmony with my current agent requirements. Dissapointed this didnt make its way through to 1.5 release. Viktor
          Hide
          Jeremy Karlson added a comment -

          This can probably use a bit more testing; I'll be doing some as well, but I welcome additional testing and feedback.

          Show
          Jeremy Karlson added a comment - This can probably use a bit more testing; I'll be doing some as well, but I welcome additional testing and feedback.

            People

            • Assignee:
              Jeremy Karlson
              Reporter:
              Jeremy Karlson
            • Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development