Flume
  1. Flume
  2. FLUME-1149

All sources get same channel list even if configuration is different.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.2.0
    • Component/s: Sinks+Sources
    • Labels:
      None
    1. FLUME-1149-1.patch
      1 kB
      Hari Shreedharan
    2. FLUME-trunk-1149.patch
      0.8 kB
      Mubarak Seyed

      Activity

      Hide
      jiraposter@reviews.apache.org added a comment -

      -----------------------------------------------------------
      This is an automatically generated e-mail. To reply, visit:
      https://reviews.apache.org/r/4872/
      -----------------------------------------------------------

      Review request for Flume.

      Summary
      -------

      Even if config specifies different files, all sources end up writing to all channels. Fixed it.

      This addresses bug FLUME-1149.
      https://issues.apache.org/jira/browse/FLUME-1149

      Diffs


      flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java c0f83e4

      Diff: https://reviews.apache.org/r/4872/diff

      Testing
      -------

      Functional testing with multiple channels.

      Thanks,

      Hari

      Show
      jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4872/ ----------------------------------------------------------- Review request for Flume. Summary ------- Even if config specifies different files, all sources end up writing to all channels. Fixed it. This addresses bug FLUME-1149 . https://issues.apache.org/jira/browse/FLUME-1149 Diffs flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java c0f83e4 Diff: https://reviews.apache.org/r/4872/diff Testing ------- Functional testing with multiple channels. Thanks, Hari
      Hide
      jiraposter@reviews.apache.org added a comment -

      -----------------------------------------------------------
      This is an automatically generated e-mail. To reply, visit:
      https://reviews.apache.org/r/4872/#review7208
      -----------------------------------------------------------

      Ship it!

      lgtm! I confirmed the fix works.

      Below is the config I used. I had to make sure to first create the /home/will/results1 and /home/will/results2 dirs before starting the agent.

      Then I started the agent, waited for each of the 2 files to be created, and tailed them in diff't terminals.

      Then I launched interactive netcat terminals in each of 2 consoles:
      nc localhost 1473
      nc localhost 1474

      Before the patch, I entered "0" into the first console going to port 1473, and the "0" was logged to both sources. After the patch, I entered "0" into the first console going to port 1473, and "0" appeared only for src1 destination. Then I entered "1" into second console going to port 1474, and "1" appeared only for src2 destination.

      1. a = agent
      2. c = channel
      3. r = source
      4. k = sink
      5. g = sink group
        agent.channels = c1 c2
        agent.sources = r1 r2
        agent.sinks = k1 k2

      agent.channels.c1.type = MEMORY
      agent.channels.c2.type = MEMORY

      agent.sources.r1.type = NETCAT
      agent.sources.r1.channels = c1
      agent.sources.r1.bind = 0.0.0.0
      agent.sources.r1.port = 1473

      agent.sources.r2.type = NETCAT
      agent.sources.r2.channels = c2
      agent.sources.r2.bind = 0.0.0.0
      agent.sources.r2.port = 1474

      agent.sinks.k1.channel = c1
      agent.sinks.k1.type = FILE_ROLL
      agent.sinks.k1.sink.directory = /home/will/results1
      agent.sinks.k1.sink.rollInterval = 0

      agent.sinks.k2.channel = c2
      agent.sinks.k2.type = FILE_ROLL
      agent.sinks.k2.sink.directory = /home/will/results2
      agent.sinks.k2.sink.rollInterval = 0

      • Will

      On 2012-04-25 09:51:43, Hari Shreedharan wrote:

      -----------------------------------------------------------

      This is an automatically generated e-mail. To reply, visit:

      https://reviews.apache.org/r/4872/

      -----------------------------------------------------------

      (Updated 2012-04-25 09:51:43)

      Review request for Flume.

      Summary

      -------

      Even if config specifies different files, all sources end up writing to all channels. Fixed it.

      This addresses bug FLUME-1149.

      https://issues.apache.org/jira/browse/FLUME-1149

      Diffs

      -----

      flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java c0f83e4

      Diff: https://reviews.apache.org/r/4872/diff

      Testing

      -------

      Functional testing with multiple channels.

      Thanks,

      Hari

      Show
      jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4872/#review7208 ----------------------------------------------------------- Ship it! lgtm! I confirmed the fix works. Below is the config I used. I had to make sure to first create the /home/will/results1 and /home/will/results2 dirs before starting the agent. Then I started the agent, waited for each of the 2 files to be created, and tailed them in diff't terminals. Then I launched interactive netcat terminals in each of 2 consoles: nc localhost 1473 nc localhost 1474 Before the patch, I entered "0" into the first console going to port 1473, and the "0" was logged to both sources. After the patch, I entered "0" into the first console going to port 1473, and "0" appeared only for src1 destination. Then I entered "1" into second console going to port 1474, and "1" appeared only for src2 destination. a = agent c = channel r = source k = sink g = sink group agent.channels = c1 c2 agent.sources = r1 r2 agent.sinks = k1 k2 agent.channels.c1.type = MEMORY agent.channels.c2.type = MEMORY agent.sources.r1.type = NETCAT agent.sources.r1.channels = c1 agent.sources.r1.bind = 0.0.0.0 agent.sources.r1.port = 1473 agent.sources.r2.type = NETCAT agent.sources.r2.channels = c2 agent.sources.r2.bind = 0.0.0.0 agent.sources.r2.port = 1474 agent.sinks.k1.channel = c1 agent.sinks.k1.type = FILE_ROLL agent.sinks.k1.sink.directory = /home/will/results1 agent.sinks.k1.sink.rollInterval = 0 agent.sinks.k2.channel = c2 agent.sinks.k2.type = FILE_ROLL agent.sinks.k2.sink.directory = /home/will/results2 agent.sinks.k2.sink.rollInterval = 0 Will On 2012-04-25 09:51:43, Hari Shreedharan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4872/ ----------------------------------------------------------- (Updated 2012-04-25 09:51:43) Review request for Flume. Summary ------- Even if config specifies different files, all sources end up writing to all channels. Fixed it. This addresses bug FLUME-1149 . https://issues.apache.org/jira/browse/FLUME-1149 Diffs ----- flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java c0f83e4 Diff: https://reviews.apache.org/r/4872/diff Testing ------- Functional testing with multiple channels. Thanks, Hari
      Hide
      jiraposter@reviews.apache.org added a comment -

      -----------------------------------------------------------
      This is an automatically generated e-mail. To reply, visit:
      https://reviews.apache.org/r/4872/#review7209
      -----------------------------------------------------------

      Ship it!

      Looks good to me, verified that a config similar to Will's does not produce the previous doubling behavior. Thanks for the fix Hari!

      • Mike

      On 2012-04-25 09:51:43, Hari Shreedharan wrote:

      -----------------------------------------------------------

      This is an automatically generated e-mail. To reply, visit:

      https://reviews.apache.org/r/4872/

      -----------------------------------------------------------

      (Updated 2012-04-25 09:51:43)

      Review request for Flume.

      Summary

      -------

      Even if config specifies different files, all sources end up writing to all channels. Fixed it.

      This addresses bug FLUME-1149.

      https://issues.apache.org/jira/browse/FLUME-1149

      Diffs

      -----

      flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java c0f83e4

      Diff: https://reviews.apache.org/r/4872/diff

      Testing

      -------

      Functional testing with multiple channels.

      Thanks,

      Hari

      Show
      jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4872/#review7209 ----------------------------------------------------------- Ship it! Looks good to me, verified that a config similar to Will's does not produce the previous doubling behavior. Thanks for the fix Hari! Mike On 2012-04-25 09:51:43, Hari Shreedharan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4872/ ----------------------------------------------------------- (Updated 2012-04-25 09:51:43) Review request for Flume. Summary ------- Even if config specifies different files, all sources end up writing to all channels. Fixed it. This addresses bug FLUME-1149 . https://issues.apache.org/jira/browse/FLUME-1149 Diffs ----- flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java c0f83e4 Diff: https://reviews.apache.org/r/4872/diff Testing ------- Functional testing with multiple channels. Thanks, Hari
      Hide
      Arvind Prabhakar added a comment -

      Patch committed. Thanks Hari!

      Show
      Arvind Prabhakar added a comment - Patch committed. Thanks Hari!
      Hide
      jiraposter@reviews.apache.org added a comment -

      -----------------------------------------------------------
      This is an automatically generated e-mail. To reply, visit:
      https://reviews.apache.org/r/4872/#review7210
      -----------------------------------------------------------

      Ship it!

      +1

      • Arvind

      On 2012-04-25 09:51:43, Hari Shreedharan wrote:

      -----------------------------------------------------------

      This is an automatically generated e-mail. To reply, visit:

      https://reviews.apache.org/r/4872/

      -----------------------------------------------------------

      (Updated 2012-04-25 09:51:43)

      Review request for Flume.

      Summary

      -------

      Even if config specifies different files, all sources end up writing to all channels. Fixed it.

      This addresses bug FLUME-1149.

      https://issues.apache.org/jira/browse/FLUME-1149

      Diffs

      -----

      flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java c0f83e4

      Diff: https://reviews.apache.org/r/4872/diff

      Testing

      -------

      Functional testing with multiple channels.

      Thanks,

      Hari

      Show
      jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4872/#review7210 ----------------------------------------------------------- Ship it! +1 Arvind On 2012-04-25 09:51:43, Hari Shreedharan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4872/ ----------------------------------------------------------- (Updated 2012-04-25 09:51:43) Review request for Flume. Summary ------- Even if config specifies different files, all sources end up writing to all channels. Fixed it. This addresses bug FLUME-1149 . https://issues.apache.org/jira/browse/FLUME-1149 Diffs ----- flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java c0f83e4 Diff: https://reviews.apache.org/r/4872/diff Testing ------- Functional testing with multiple channels. Thanks, Hari
      Hide
      Hudson added a comment -

      Integrated in flume-trunk #184 (See https://builds.apache.org/job/flume-trunk/184/)
      FLUME-1149. All sources get same channel list even if configuration is different.

      (Hari Shreedharan via Arvind Prabhakar) (Revision 1330222)

      Result = SUCCESS
      arvind : http://svn.apache.org/viewvc/?view=rev&rev=1330222
      Files :

      • /incubator/flume/trunk/flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java
      Show
      Hudson added a comment - Integrated in flume-trunk #184 (See https://builds.apache.org/job/flume-trunk/184/ ) FLUME-1149 . All sources get same channel list even if configuration is different. (Hari Shreedharan via Arvind Prabhakar) (Revision 1330222) Result = SUCCESS arvind : http://svn.apache.org/viewvc/?view=rev&rev=1330222 Files : /incubator/flume/trunk/flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java
      Hide
      Mubarak Seyed added a comment -

      The attached file (FLUME-trunk-1149.patch) is a patch for multiplexing selector. Ran a unit test and tested in lab environment.

      Thanks.

      Show
      Mubarak Seyed added a comment - The attached file (FLUME-trunk-1149.patch) is a patch for multiplexing selector. Ran a unit test and tested in lab environment. Thanks.
      Hide
      Mike Percy added a comment -

      Mubarak, thanks for the patch!

      Please open a new JIRA for this issue and post to review board separately.

      Thanks, Mike

      Show
      Mike Percy added a comment - Mubarak, thanks for the patch! Please open a new JIRA for this issue and post to review board separately. Thanks, Mike
      Hide
      Mike Percy added a comment -

      Oh, it's a one-line patch, maybe just another JIRA is fine and RB is not really necessary

      Show
      Mike Percy added a comment - Oh, it's a one-line patch, maybe just another JIRA is fine and RB is not really necessary
      Hide
      Mubarak Seyed added a comment -

      Thanks Mike. I will open a JIRA and attach a patch. Thanks.

      Show
      Mubarak Seyed added a comment - Thanks Mike. I will open a JIRA and attach a patch. Thanks.
      Hide
      Mubarak Seyed added a comment -

      FLUME-1180 created for a patch. Thanks.

      Show
      Mubarak Seyed added a comment - FLUME-1180 created for a patch. Thanks.

        People

        • Assignee:
          Hari Shreedharan
          Reporter:
          Hari Shreedharan
        • Votes:
          0 Vote for this issue
          Watchers:
          0 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development