Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1379

avro source , hdfs sink, no event writed to hdfs

    XMLWordPrintableJSON

    Details

    • Type: Question
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0, 1.3.0
    • Fix Version/s: 1.3.0
    • Component/s: Sinks+Sources
    • Labels:
    • Environment:

      rhel5.7 ,jdk1.6,yum installed flume-ng 1.1.0+120-1.cdh4.0.0 ,mvn install 1.3.

      Description

      hello everyone!
      1, source's type is exec and sink's type is hdfs .the event can write to the hdfs.But when I change the source's type to avro ,no event write to hdfs.
      2, flume.conf
      #List sources, sinks and channels in the agent
      weblog-agent.sources = s1
      weblog-agent.sinks = avro-forward-sink01
      weblog-agent.channels = jdbc-channel01
      #define the flow
      #webblog-agent sources config
      weblog-agent.sources.s1.channels = jdbc-channel01
      weblog-agent.sources.s1.type = avro
      weblog-agent.sources.s1.port = 41414
      weblog-agent.sources.s1.bind = 0.0.0.0
      #avro sink properties
      weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01
      weblog-agent.sinks.avro-forward-sink01.type = hdfs
      weblog-agent.sinks.avro-forward-sink01.hdfs.path = hdfs://hadoop-master.:9000/user/flume/webtest
      #weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_%p
      #weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_web001
      weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosPrincipal = flume@HADOOP.COM
      weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosKeytab = /var/run/flume-ng/flume.keytab
      weblog-agent.sinks.avro-forward-sink01.hdfs.maxOpenFiles = 65535
      weblog-agent.sinks.avro-forward-sink01.hdfs.rollSize = 0
      weblog-agent.sinks.avro-forward-sink01.hdfs.rollCount = 0
      weblog-agent.sinks.avro-forward-sink01.hdfs.batchSize = 30000
      weblog-agent.sinks.avro-forward-sink01.hdfs.txnEventMax = 20000
      weblog-agent.sinks.avro-forward-sink01.hdfs.fileType = DataStream
      #channels config
      weblog-agent.channels.jdbc-channel01.type = memory
      weblog-agent.channels.jdbc-channel01.capacity = 200000
      weblog-agent.channels.jdbc-channel01.transactionCapacity = 20000

      3,flume.log

      2012-07-18 23:53:31,245 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1
      2012-07-18 23:53:31,247 INFO node.FlumeNode: Flume node starting - weblog-agent
      2012-07-18 23:53:31,251 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting
      2012-07-18 23:53:31,252 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 10
      2012-07-18 23:53:31,252 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting
      2012-07-18 23:53:31,252 DEBUG nodemanager.DefaultLogicalNodeManager: Node manager started
      2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: Configuration provider started
      2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider: Checking file:flume.conf for changes
      2012-07-18 23:53:31,254 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:flume.conf
      2012-07-18 23:53:31,261 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
      2012-07-18 23:53:31,261 DEBUG conf.FlumeConfiguration: Created context for avro-forward-sink01: hdfs.fileType
      2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
      2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
      2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
      2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
      2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
      2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
      2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Added sinks: avro-forward-sink01 Agent: weblog-agent
      2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
      2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
      2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
      2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Processing:avro-forward-sink01
      2012-07-18 23:53:31,263 DEBUG conf.FlumeConfiguration: Starting validation of configuration for agent: weblog-agent, initial-configuration: AgentConfiguration[weblog-agent]
      SOURCES: {s1={ parameters:

      {port=41414, channels=jdbc-channel01, type=avro, bind=0.0.0.0} }}
      CHANNELS: {jdbc-channel01={ parameters:{transactionCapacity=20000, capacity=200000, type=memory} }}
      SINKS: {avro-forward-sink01={ parameters:{hdfs.fileType=DataStream, hdfs.kerberosPrincipal=flume@HADOOP.COM, hdfs.txnEventMax=20000, hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab, hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest, hdfs.batchSize=30000, hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs, channel=jdbc-channel01, hdfs.rollCount=0} }}

      2012-07-18 23:53:31,271 DEBUG conf.FlumeConfiguration: Created channel jdbc-channel01
      2012-07-18 23:53:31,289 DEBUG conf.FlumeConfiguration: Creating sink: avro-forward-sink01 using HDFS
      2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Post validation configuration for weblog-agent
      AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[weblog-agent]
      SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, type=avro, bind=0.0.0.0}

      }}
      CHANNELS: {jdbc-channel01={ parameters:

      {transactionCapacity=20000, capacity=200000, type=memory}

      }}
      SINKS: {avro-forward-sink01={ parameters:

      {hdfs.fileType=DataStream, hdfs.kerberosPrincipal=flume@HADOOP.COM, hdfs.txnEventMax=20000, hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab, hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest, hdfs.batchSize=30000, hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs, channel=jdbc-channel01, hdfs.rollCount=0}

      }}

      2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Channels:jdbc-channel01

      2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Sinks avro-forward-sink01

      2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Sources s1

      2012-07-18 23:53:31,292 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [weblog-agent]
      2012-07-18 23:53:31,292 INFO properties.PropertiesFileConfigurationProvider: Creating channels
      2012-07-18 23:53:31,292 DEBUG channel.DefaultChannelFactory: Creating instance of channel jdbc-channel01 type memory
      2012-07-18 23:53:31,365 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: jdbc-channel01, registered successfully.
      2012-07-18 23:53:31,365 INFO properties.PropertiesFileConfigurationProvider: created channel jdbc-channel01
      2012-07-18 23:53:31,365 DEBUG source.DefaultSourceFactory: Creating instance of source s1, type avro
      2012-07-18 23:53:31,380 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SOURCE, name: s1, registered successfully.
      2012-07-18 23:53:31,395 INFO sink.DefaultSinkFactory: Creating instance of sink: avro-forward-sink01, type: hdfs
      2012-07-18 23:53:31,540 DEBUG security.Groups: Creating new Groups object
      2012-07-18 23:53:31,625 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
      2012-07-18 23:53:31,640 INFO hdfs.HDFSEventSink: Hadoop Security enabled: true
      2012-07-18 23:53:31,789 INFO hdfs.HDFSEventSink:

      { Sink type:HDFSEventSink, name:avro-forward-sink01 }

      : Attempting kerberos login as principal (flume@HADOOP.COM) from keytab file (/var/run/flume-ng/flume.keytab)
      2012-07-18 23:53:31,999 INFO security.UserGroupInformation: Login successful for user flume@HADOOP.COM using keytab file /var/run/flume-ng/flume.keytab
      2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Auth method: KERBEROS
      2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: User name: flume@HADOOP.COM
      2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Using keytab: true
      2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Logged in as user flume@HADOOP.COM
      2012-07-18 23:53:32,002 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: avro-forward-sink01, registered successfully.
      2012-07-18 23:53:32,004 INFO nodemanager.DefaultLogicalNodeManager: Starting new configuration:{ sourceRunners:{s1=EventDrivenSourceRunner: { source:Avro source s1:

      { bindAddress: 0.0.0.0, port: 41414 }

      }} sinkRunners:{avro-forward-sink01=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6c1989b5 counterGroup:{ name:null counters:{} } }} channels:

      {jdbc-channel01=org.apache.flume.channel.MemoryChannel@a00185}

      }
      2012-07-18 23:53:32,004 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel jdbc-channel01
      2012-07-18 23:53:32,005 INFO nodemanager.DefaultLogicalNodeManager: Waiting for channel: jdbc-channel01 to start. Sleeping for 500 ms
      2012-07-18 23:53:32,005 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: jdbc-channel01 started
      2012-07-18 23:53:32,507 INFO nodemanager.DefaultLogicalNodeManager: Starting Sink avro-forward-sink01
      2012-07-18 23:53:32,508 INFO nodemanager.DefaultLogicalNodeManager: Starting Source s1
      2012-07-18 23:53:32,508 INFO source.AvroSource: Starting Avro source s1:

      { bindAddress: 0.0.0.0, port: 41414 }

      ...
      2012-07-18 23:53:32,511 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: avro-forward-sink01 started
      2012-07-18 23:53:32,514 DEBUG flume.SinkRunner: Polling sink runner starting
      2012-07-18 23:53:32,941 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: s1 started
      2012-07-18 23:53:32,941 INFO source.AvroSource: Avro source s1 started.
      2012-07-18 23:54:02,514 DEBUG properties.PropertiesFileConfigurationProvider: Checking file:flume.conf for changes
      2012-07-18 23:54:32,521 DEBUG properties.PropertiesFileConfigurationProvider: Checking file:flume.conf for changes

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              lqlboy jack li
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 504h
                504h
                Remaining:
                Remaining Estimate - 504h
                504h
                Logged:
                Time Spent - Not Specified
                Not Specified