[FLUME-2826] Right Configuration of flume agent for Twitter data

• Type: Blog - New Blog Request
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.5.2
• Fix Version/s: 1.5.2
• Component/s: Configuration
• Assignee: Unassigned
• Reporter: Kettler Karl
• Created: 23/Oct/15 08:25
• Updated: 21/Jun/16 07:43
• Resolved: 21/Jun/16 07:43

Description

How does you get Keywords and language?

TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = xxx
TwitterAgent.sources.Twitter.consumerSecret = xxx
TwitterAgent.sources.Twitter.accessToken = xxx
TwitterAgent.sources.Twitter.accessTokenSecret = xxx
TwitterAgent.sources.Twitter.maxBatchSize = 10
TwitterAgent.sources.Twitter.maxBatchDurationMillis = 200
TwitterAgent.sources.Twitter.keywords = United Nations
TwitterAgent.sources.Twitter.deserializer.schemaType = LITERAL
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = /demo/tweets/stream/%y-%m-%d/%H%M%S
TwitterAgent.sinks.HDFS.hdfs.filePrefix = events
TwitterAgent.sinks.HDFS.hdfs.round = true
TwitterAgent.sinks.HDFS.hdfs.roundValue = 5
TwitterAgent.sinks.HDFS.hdfs.roundUnit = minute
TwitterAgent.sinks.HDFS.hdfs.useLocalTimeStamp = true
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 1000
TwitterAgent.channels.MemChannel.transactionCapacity = 100

Kind regards,
Karl