Flume
  1. Flume
  2. FLUME-299

Add support for subarguments in flume config elements

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v0.9.4
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      It would make sense for certain arguments to have sub-arguments.

      For example, when specifiying the console("avro") sink, it would be nice to give the avro output format arguments. So we could be able to say 'console(avro("json"))' or 'console(avro("bin","gzip"))'

      This would also allow roll sinks to specify different trigger policies:

      'roll( timer(1000))

      { ... }' // roll every second
      'roll( count(100)) { ... }

      ' // roll every 100 events
      'roll( countTimer(100,1000))

      { ... }

      ' // roll every 100 events or every 1000 ms
      'roll( size(1000000) )

      {... }

      ' // roll every 1000000 bytes.

      It would also allow failovers sinks to specify retry policies.

        Issue Links

          Activity

          Jonathan Hsieh made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Mark Thomas made changes -
          Project Import Tue Aug 02 16:57:12 UTC 2011 [ 1312304232406 ]
          Jonathan Hsieh made changes -
          Link This issue blocks FLUME-541 [ FLUME-541 ]
          Jonathan Hsieh made changes -
          Link This issue relates to FLUME-472 [ FLUME-472 ]
          Jonathan Hsieh made changes -
          Status Patch Available [ 10000 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Fix Version/s v0.9.4 [ 10050 ]
          Hide
          Jonathan Hsieh added a comment -

          committed.

          Show
          Jonathan Hsieh added a comment - committed.
          Hide
          Jonathan Hsieh added a comment -

          Pretty sure it was Bruce.

          I prefer a leaner, terser language, but can understand the desire for json or some other pre-existing scripting language's syntax (a lispish sexp was suggested as well). We essentially emulate java for our arguments and value syntax.

          I wouldn't oppose an alternative (maybe for a 2.0 flume), but my main thought is that we have something now that works, and many other higher priority issues to deal with to get to a 1.0 flume.

          Show
          Jonathan Hsieh added a comment - Pretty sure it was Bruce. I prefer a leaner, terser language, but can understand the desire for json or some other pre-existing scripting language's syntax (a lispish sexp was suggested as well). We essentially emulate java for our arguments and value syntax. I wouldn't oppose an alternative (maybe for a 2.0 flume), but my main thought is that we have something now that works, and many other higher priority issues to deal with to get to a 1.0 flume.
          Hide
          E. Sammer added a comment -

          I completely agree with the need to be able to provide richer args to sources and sinks. I know that someone, at some point, proposed just moving to JSON as a configuration language. I'm not sure how I like that either (given the need to be able to do strong, shell side validation of parameters, etc.) but it's probably an option worth considering. I don't want to block this, but it looks like we're getting closer and closer to JSON; should we just make the jump? The complexity of this starts to feel hinky.

          Show
          E. Sammer added a comment - I completely agree with the need to be able to provide richer args to sources and sinks. I know that someone, at some point, proposed just moving to JSON as a configuration language. I'm not sure how I like that either (given the need to be able to do strong, shell side validation of parameters, etc.) but it's probably an option worth considering. I don't want to block this, but it looks like we're getting closer and closer to JSON; should we just make the jump? The complexity of this starts to feel hinky.
          Jonathan Hsieh made changes -
          Hide
          Jonathan Hsieh added a comment -
          Show
          Jonathan Hsieh added a comment - review here https://review.cloudera.org/r/1757/
          Jonathan Hsieh made changes -
          Status Open [ 1 ] Patch Available [ 10000 ]
          Jonathan Hsieh made changes -
          Status Patch Available [ 10000 ] Open [ 1 ]
          Jonathan Hsieh made changes -
          Assignee Jonathan Hsieh [ jmhsieh ]
          Jonathan Hsieh made changes -
          Status Open [ 1 ] Patch Available [ 10000 ]
          Jonathan Hsieh made changes -
          Link This issue blocks FLUME-632 [ FLUME-632 ]
          Jonathan Hsieh made changes -
          Summary Add support for arguments flume config elements Add support for subarguments in flume config elements
          Jonathan Hsieh made changes -
          Link This issue relates to FLUME-133 [ FLUME-133 ]
          Jonathan Hsieh made changes -
          Field Original Value New Value
          Link This issue blocks FLUME-273 [ FLUME-273 ]
          Jonathan Hsieh created issue -

            People

            • Assignee:
              Jonathan Hsieh
              Reporter:
              Jonathan Hsieh
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development