Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2826

Right Configuration of flume agent for Twitter data

    XMLWordPrintableJSON

    Details

    • Type: Blog - New Blog Request
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 1.5.2
    • Fix Version/s: 1.5.2
    • Component/s: Configuration
    • Labels:
      None

      Description

      How does you get Keywords and language?

      TwitterAgent.sources = Twitter
      TwitterAgent.channels = MemChannel
      TwitterAgent.sinks = HDFS
      TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
      TwitterAgent.sources.Twitter.channels = MemChannel
      TwitterAgent.sources.Twitter.consumerKey = xxx
      TwitterAgent.sources.Twitter.consumerSecret = xxx
      TwitterAgent.sources.Twitter.accessToken = xxx
      TwitterAgent.sources.Twitter.accessTokenSecret = xxx
      TwitterAgent.sources.Twitter.maxBatchSize = 10
      TwitterAgent.sources.Twitter.maxBatchDurationMillis = 200
      TwitterAgent.sources.Twitter.keywords = United Nations
      TwitterAgent.sources.Twitter.deserializer.schemaType = LITERAL
      TwitterAgent.sinks.HDFS.channel = MemChannel
      TwitterAgent.sinks.HDFS.type = hdfs
      TwitterAgent.sinks.HDFS.hdfs.path = /demo/tweets/stream/%y-%m-%d/%H%M%S
      TwitterAgent.sinks.HDFS.hdfs.filePrefix = events
      TwitterAgent.sinks.HDFS.hdfs.round = true
      TwitterAgent.sinks.HDFS.hdfs.roundValue = 5
      TwitterAgent.sinks.HDFS.hdfs.roundUnit = minute
      TwitterAgent.sinks.HDFS.hdfs.useLocalTimeStamp = true
      TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
      TwitterAgent.channels.MemChannel.type = memory
      TwitterAgent.channels.MemChannel.capacity = 1000
      TwitterAgent.channels.MemChannel.transactionCapacity = 100

      Kind regards,
      Karl

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Karl24 Kettler Karl
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: