Flume
  1. Flume
  2. FLUME-1004

The SEQ source doesn't seem to stop on a reconfig that should disable it

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: v1.1.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      RHEL 6.1 64-bit

      Description

      Steps:

      1) Run the flume node with a config file that has the sink disabled:

      a1.sources = r1
      a1.channels = c1
      #a1.sinks = k1
      a1.sources.r1.type = SEQ
      a1.sources.r1.channels = c1
      a1.channels.c1.type = MEMORY
      #a1.sinks.k1.type = logger
      #a1.sinks.k1.channel = c1

      If you add a print statement to ChannelProcessor.processEvent(), you'll see that the source is sending events to the channel.

      2) Modify the config so that the source is disabled, and a new sink is enabled:

      #a1.sources = r1
      a1.channels = c1
      a1.sinks = k1
      #a1.sources.r1.type = SEQ
      #a1.sources.r1.channels = c1
      a1.channels.c1.type = MEMORY
      a1.sinks.k1.type = logger
      a1.sinks.k1.channel = c1

      //exp: The sink should only receive events that were in the channel at the time that the source was enabled (effectively draining the source)
      //act: The sink continuously receives events from the same source that is disabled according to the config, but is enabled according to the continuous output seen.

        Issue Links

          Activity

          Hide
          Arvind Prabhakar added a comment -

          I suggest waiting for FLUME-968 to be addressed first.

          Show
          Arvind Prabhakar added a comment - I suggest waiting for FLUME-968 to be addressed first.
          Hide
          Juhani Connolly added a comment -

          If no-one else is interested I can look into fixing reconfiguration. A bit busy for the next couple of days, but could start on it around Friday.

          That being said, before that, it would be a good idea to take all the non-file specific configuration out of PropertiesFileConfigurationProvider

          Show
          Juhani Connolly added a comment - If no-one else is interested I can look into fixing reconfiguration. A bit busy for the next couple of days, but could start on it around Friday. That being said, before that, it would be a good idea to take all the non-file specific configuration out of PropertiesFileConfigurationProvider
          Hide
          Arvind Prabhakar added a comment -

          @Juhani - you are right, that is how the system is implemented at this stage. However, this seems to be more of technical debt rather than design choice.

          Shutting down of components on configuration change is needed in order to make sure that there are no resource leaks in the system and that these components can be dynamically reconfigured during runtime.

          Show
          Arvind Prabhakar added a comment - @Juhani - you are right, that is how the system is implemented at this stage. However, this seems to be more of technical debt rather than design choice. Shutting down of components on configuration change is needed in order to make sure that there are no resource leaks in the system and that these components can be dynamically reconfigured during runtime.
          Hide
          Juhani Connolly added a comment - - edited

          correct me if I'm wrong, but I don't think reconfiguration shuts down anything that is removed from the conf?

          Reconfiguration is currently instantiated by AbstractFileConfiguration's FileWatcherRunnable which just calls doLoad() on the configuration provider if the conf has changed.

          doLoad in turn validates the current configuration when it is loaded by FlumeConfiguration, and from there goes on to load channels/sources/sinks(the currently running ones are retrieved by the factory methods). The configure call results in reconfiguring the currently running components.
          At no point are old channels/sources/sinks stopped.

          The current method is pretty neat but we're going to have to add a mechanism somewhere to shutdown disabled components. Perhaps this should be added to onNodeConfigurationChanged?

          Show
          Juhani Connolly added a comment - - edited correct me if I'm wrong, but I don't think reconfiguration shuts down anything that is removed from the conf? Reconfiguration is currently instantiated by AbstractFileConfiguration's FileWatcherRunnable which just calls doLoad() on the configuration provider if the conf has changed. doLoad in turn validates the current configuration when it is loaded by FlumeConfiguration, and from there goes on to load channels/sources/sinks(the currently running ones are retrieved by the factory methods). The configure call results in reconfiguring the currently running components. At no point are old channels/sources/sinks stopped. The current method is pretty neat but we're going to have to add a mechanism somewhere to shutdown disabled components. Perhaps this should be added to onNodeConfigurationChanged?

            People

            • Assignee:
              Unassigned
              Reporter:
              Will McQueen
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development