[FLUME-167] Flow isolation (multiple flows case) + user guide correction request

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 0.9.2

Environment

ubuntu 10.04+cdh3

• Assignee: Jonathan Hsieh
• Reporter: Disabled imported user
• Created: 18/Aug/10 02:11
• Updated: 13/Dec/10 14:38
• Resolved: 17/Sep/10 07:46

Description

Hi flume team,

1. multiple flows isolation
As the post (https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/79051d79fb114f82/29085e233aed8bf8?lnk=gst&q=flow#29085e233aed8bf8) in the forum suggested, We used the 'config' command to make the flow isolation work for a single flow case. However, when it deals with more than 1 flows, all the data will go to the same hdfs sinks instead of going to the different sinks as the collectors suggest. Here is our config:

Flows:
fl.flow: agent1->collector1->hdfs://place 1
f2.flow: agent3->collector2->hdfs://place 2
physical nodes: sftrash, collector. Both on the same box.

Setting: (grabbed from master page)
86 SUCCEEDED config [agent1, f1.flow, tail("/var/log/tomcat6/oozie.log"), autoBEChain]
87 SUCCEEDED config [collector1, f1.flow, autoCollectorSource, collectorSink( "hdfs://localhost/user/hadoop/flumelog", "_flume_", 30000)]
88 SUCCEEDED config [agent3, f2.flow, tail("/var/log/apache2/access.log"), autoBEChain]
89 SUCCEEDED config [collector2, f2.flow, autoCollectorSource, collectorSink( "hdfs://localhost/user/hadoop/flumelog", "_log_", 30000)]
94 SUCCEEDED spawn [collector, collector2]
95 SUCCEEDED spawn [collector, collector1]
96 SUCCEEDED spawn [sftrash, agent1]
97 SUCCEEDED spawn [sftrash, agent3]

Translated sink for agent1:
{ lazyOpen =>

{ stubbornAppend => rpcSink( "ubuntu.ubuntu-domain", 35859 ) }

}
Translated sink for agent3:
{ lazyOpen =>

{ stubbornAppend => rpcSink( "ubuntu.ubuntu-domain", 35858 ) }

}
Translated source for collector1
rpcSource( 35859 )
Translated source for collector2
rpcSource( 35858 )

Both sinks and sources for physical nodes sftrash and collector are NULL.

Since the translation can assign different rpc source ports for those two collectors, I guess maybe I need specify the physical nodes or spawn in another way?

2. user guide correction request
As Henry's request, the typo for "spawn" in flow isolation can be found by searching "spawn host3 barConsumer flowbar" in the user guide Jen 29, 2010 version.

Thanks a lot!

Wes

Issue Links

• duplicates: FLUME-151 Commands in Flow Isolation section are incorrect.