Flume
  1. Flume
  2. FLUME-1000

Properties validator misses an invalid config

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.1.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      RHEL 6.2 64-bit

      Description

      The following config is invalid, but is not caught by the properties validator (no sinks are specified).

      a1.sources = r1
      a1.channels = c1
      #a1.sinks = k1
      a1.sources.r1.type = SEQ
      a1.sources.r1.channels = c1
      a1.channels.c1.type = MEMORY
      #a1.sinks.k1.type = logger
      #a1.sinks.k1.channel = c1

      If I run with this config, then the source and channel instance is created. If you add a print statement to ChannelProcessor.processEvent(), you'll see that the source is sending events to the channel. The source generates about 104 events, and then generates an additional one about every second... which is interesting, since the mem channel's default capacity is only 100 and there's no sink. When I do a runtime configuration by changing the the props file to this:

      #a1.sources = r1
      a1.channels = c1
      a1.sinks = k1
      #a1.sources.r1.type = SEQ
      #a1.sources.r1.channels = c1
      a1.channels.c1.type = MEMORY
      a1.sinks.k1.type = logger
      a1.sinks.k1.channel = c1

      ...then sink instance is created but the SEQ source is not stopped – it continuously sends events at a rapid pace.

      I mention this because observing the behavior of even invalid configs might give clues as to problems that might occur with valid configs (such as a source not being stopped upon dynamic reconfig).

        Activity

        Hide
        Arvind Prabhakar added a comment -

        @Will - this is not an invalid config. It is perfectly fine for an agent to not have a sink configured. Similarly it is perfectly fine to have a channel and sink without a source for an Agent in order to drain the channel.

        Show
        Arvind Prabhakar added a comment - @Will - this is not an invalid config. It is perfectly fine for an agent to not have a sink configured. Similarly it is perfectly fine to have a channel and sink without a source for an Agent in order to drain the channel.
        Hide
        Arvind Prabhakar added a comment -

        Also, I think we should have separate jiras to track

        • That the memory channel does seem to respect the capacity
        • That the sequence file source does not seem to stop on reconfiguration that disables it.
        Show
        Arvind Prabhakar added a comment - Also, I think we should have separate jiras to track That the memory channel does seem to respect the capacity That the sequence file source does not seem to stop on reconfiguration that disables it.
        Hide
        Will McQueen added a comment - - edited

        Thanks, Arvind. The javadocs for PropertiesFileConfigurationProvider say:
        "For every <agent name> there must be three lists specified that include <agent name>.sources, <agent name>.sinks, and <agent name>.channels. Each of these lists must contain a space separated list of names corresponding to that particular entity."

        I assumed this list must be non-empty, but that assumption was wrong.

        Show
        Will McQueen added a comment - - edited Thanks, Arvind. The javadocs for PropertiesFileConfigurationProvider say: "For every <agent name> there must be three lists specified that include <agent name>.sources, <agent name>.sinks, and <agent name>.channels. Each of these lists must contain a space separated list of names corresponding to that particular entity." I assumed this list must be non-empty, but that assumption was wrong.
        Hide
        Arvind Prabhakar added a comment -

        Good point Will. We can use this issue to track javadoc changes to ensure that this is clarified.

        Show
        Arvind Prabhakar added a comment - Good point Will. We can use this issue to track javadoc changes to ensure that this is clarified.
        Hide
        Will McQueen added a comment -

        Filed separate bugs per Arvinds suggestion: FLUME-1003, FLUME-1004

        Show
        Will McQueen added a comment - Filed separate bugs per Arvinds suggestion: FLUME-1003 , FLUME-1004

          People

          • Assignee:
            Unassigned
            Reporter:
            Will McQueen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development