Details
-
Question
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.1.0, 1.3.0
-
rhel5.7 ,jdk1.6,yum installed flume-ng 1.1.0+120-1.cdh4.0.0 ,mvn install 1.3.
Description
hello everyone!
1, source's type is exec and sink's type is hdfs .the event can write to the hdfs.But when I change the source's type to avro ,no event write to hdfs.
2, flume.conf
#List sources, sinks and channels in the agent
weblog-agent.sources = s1
weblog-agent.sinks = avro-forward-sink01
weblog-agent.channels = jdbc-channel01
#define the flow
#webblog-agent sources config
weblog-agent.sources.s1.channels = jdbc-channel01
weblog-agent.sources.s1.type = avro
weblog-agent.sources.s1.port = 41414
weblog-agent.sources.s1.bind = 0.0.0.0
#avro sink properties
weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01
weblog-agent.sinks.avro-forward-sink01.type = hdfs
weblog-agent.sinks.avro-forward-sink01.hdfs.path = hdfs://hadoop-master.:9000/user/flume/webtest
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_%p
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_web001
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosPrincipal = flume@HADOOP.COM
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosKeytab = /var/run/flume-ng/flume.keytab
weblog-agent.sinks.avro-forward-sink01.hdfs.maxOpenFiles = 65535
weblog-agent.sinks.avro-forward-sink01.hdfs.rollSize = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.rollCount = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.batchSize = 30000
weblog-agent.sinks.avro-forward-sink01.hdfs.txnEventMax = 20000
weblog-agent.sinks.avro-forward-sink01.hdfs.fileType = DataStream
#channels config
weblog-agent.channels.jdbc-channel01.type = memory
weblog-agent.channels.jdbc-channel01.capacity = 200000
weblog-agent.channels.jdbc-channel01.transactionCapacity = 20000
3,flume.log
2012-07-18 23:53:31,245 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1
2012-07-18 23:53:31,247 INFO node.FlumeNode: Flume node starting - weblog-agent
2012-07-18 23:53:31,251 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting
2012-07-18 23:53:31,252 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 10
2012-07-18 23:53:31,252 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting
2012-07-18 23:53:31,252 DEBUG nodemanager.DefaultLogicalNodeManager: Node manager started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: Configuration provider started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: Checking file:flume.conf for changes
2012-07-18 23:53:31,254 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:flume.conf
2012-07-18 23:53:31,261 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
2012-07-18 23:53:31,261 DEBUG conf.FlumeConfiguration: Created context for avro-forward-sink01: hdfs.fileType
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Added sinks: avro-forward-sink01 Agent: weblog-agent
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
2012-07-18 23:53:31,263 DEBUG conf.FlumeConfiguration: Starting validation of configuration for agent: weblog-agent, initial-configuration: AgentConfiguration[weblog-agent]
SOURCES: {s1={ parameters:
CHANNELS: {jdbc-channel01={ parameters:{transactionCapacity=20000, capacity=200000, type=memory} }}
SINKS: {avro-forward-sink01={ parameters:{hdfs.fileType=DataStream, hdfs.kerberosPrincipal=flume@HADOOP.COM, hdfs.txnEventMax=20000, hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab, hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest, hdfs.batchSize=30000, hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs, channel=jdbc-channel01, hdfs.rollCount=0} }}
2012-07-18 23:53:31,271 DEBUG conf.FlumeConfiguration: Created channel jdbc-channel01
2012-07-18 23:53:31,289 DEBUG conf.FlumeConfiguration: Creating sink: avro-forward-sink01 using HDFS
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Post validation configuration for weblog-agent
AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[weblog-agent]
SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, type=avro, bind=0.0.0.0}
}}
CHANNELS: {jdbc-channel01={ parameters:
}}
SINKS: {avro-forward-sink01={ parameters:
}}
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Channels:jdbc-channel01
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Sinks avro-forward-sink01
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Sources s1
2012-07-18 23:53:31,292 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [weblog-agent]
2012-07-18 23:53:31,292 INFO properties.PropertiesFileConfigurationProvider: Creating channels
2012-07-18 23:53:31,292 DEBUG channel.DefaultChannelFactory: Creating instance of channel jdbc-channel01 type memory
2012-07-18 23:53:31,365 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: jdbc-channel01, registered successfully.
2012-07-18 23:53:31,365 INFO properties.PropertiesFileConfigurationProvider: created channel jdbc-channel01
2012-07-18 23:53:31,365 DEBUG source.DefaultSourceFactory: Creating instance of source s1, type avro
2012-07-18 23:53:31,380 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SOURCE, name: s1, registered successfully.
2012-07-18 23:53:31,395 INFO sink.DefaultSinkFactory: Creating instance of sink: avro-forward-sink01, type: hdfs
2012-07-18 23:53:31,540 DEBUG security.Groups: Creating new Groups object
2012-07-18 23:53:31,625 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
2012-07-18 23:53:31,640 INFO hdfs.HDFSEventSink: Hadoop Security enabled: true
2012-07-18 23:53:31,789 INFO hdfs.HDFSEventSink:
: Attempting kerberos login as principal (flume@HADOOP.COM) from keytab file (/var/run/flume-ng/flume.keytab)
2012-07-18 23:53:31,999 INFO security.UserGroupInformation: Login successful for user flume@HADOOP.COM using keytab file /var/run/flume-ng/flume.keytab
2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Auth method: KERBEROS
2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: User name: flume@HADOOP.COM
2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Using keytab: true
2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Logged in as user flume@HADOOP.COM
2012-07-18 23:53:32,002 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: avro-forward-sink01, registered successfully.
2012-07-18 23:53:32,004 INFO nodemanager.DefaultLogicalNodeManager: Starting new configuration:{ sourceRunners:{s1=EventDrivenSourceRunner: { source:Avro source s1:
}} sinkRunners:{avro-forward-sink01=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6c1989b5 counterGroup:{ name:null counters:{} } }} channels:
{jdbc-channel01=org.apache.flume.channel.MemoryChannel@a00185} }
2012-07-18 23:53:32,004 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel jdbc-channel01
2012-07-18 23:53:32,005 INFO nodemanager.DefaultLogicalNodeManager: Waiting for channel: jdbc-channel01 to start. Sleeping for 500 ms
2012-07-18 23:53:32,005 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: jdbc-channel01 started
2012-07-18 23:53:32,507 INFO nodemanager.DefaultLogicalNodeManager: Starting Sink avro-forward-sink01
2012-07-18 23:53:32,508 INFO nodemanager.DefaultLogicalNodeManager: Starting Source s1
2012-07-18 23:53:32,508 INFO source.AvroSource: Starting Avro source s1:
...
2012-07-18 23:53:32,511 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: avro-forward-sink01 started
2012-07-18 23:53:32,514 DEBUG flume.SinkRunner: Polling sink runner starting
2012-07-18 23:53:32,941 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: s1 started
2012-07-18 23:53:32,941 INFO source.AvroSource: Avro source s1 started.
2012-07-18 23:54:02,514 DEBUG properties.PropertiesFileConfigurationProvider: Checking file:flume.conf for changes
2012-07-18 23:54:32,521 DEBUG properties.PropertiesFileConfigurationProvider: Checking file:flume.conf for changes