Details
-
Question
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.7.0
-
None
-
None
-
None
Description
I was trying to stream twitter data in HDFS using flume.Succeffuly configured agent but the data population in HDFS doesnt seems proper please help.
flume agent config:
TwitterAgent.sources= Twitter
TwitterAgent.channels= MemChannel
TwitterAgent.sinks=HDFS
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels=MemChannel
TwitterAgent.sources.Twitter.consumerKey=xxxxx
TwitterAgent.sources.Twitter.consumerSecret=xxxx
TwitterAgent.sources.Twitter.accessToken=xxxx
TwitterAgent.sources.Twitter.accessTokenSecret= xxxx
TwitterAgent.sinks.HDFS.channel=MemChannel
TwitterAgent.sinks.HDFS.type=hdfs
TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:8020/user/xxx/flume/events
TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream
TwitterAgent.sinks.HDFS.hdfs.writeformat=Text
TwitterAgent.sinks.HDFS.hdfs.batchSize=1000
TwitterAgent.sinks.HDFS.hdfs.rollSize=0
TwitterAgent.sinks.HDFS.hdfs.rollCount=10000
TwitterAgent.sinks.HDFS.hdfs.rollInterval=600
TwitterAgent.channels.MemChannel.type=memory
TwitterAgent.channels.MemChannel.capacity=10000
TwitterAgent.channels.MemChannel.transactionCapacity=100
=======================Twitter file in HDFS==============
Objavro.schema�
{"type":"record","name":"Doc","doc":"adoc","fields":[
,
{"name":"user_friends_count","type":["int","null"]},
{"name":"user_location","type":["string","null"]},
{"name":"user_description","type":["string","null"]},
{"name":"user_statuses_count","type":["int","null"]},
{"name":"user_followers_count","type":["int","null"]},
{"name":"user_name","type":["string","null"]},
{"name":"user_screen_name","type":["string","null"]},
{"name":"created_at","type":["string","null"]},
{"name":"text","type":["string","null"]},
{"name":"retweet_count","type":["long","null"]},
{"name":"retweeted","type":["boolean","null"]},
{"name":"in_reply_to_user_id","type":["long","null"]},
{"name":"source","type":["string","null"]},
{"name":"in_reply_to_status_id","type":["long","null"]},
{"name":"media_url_https","type":["string","null"]},
{"name":"expanded_url","type":["string","null"]}]}��rm���R��d�q�#�$838697771628544001Jぽやしみ��F$おやすみ世界eZ_XL1200C(2017-03-06T05:28:25Z�30ゲームから頑張ってはめ続けたのにやめて https://t.co/SmtQDJHAdz�<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>^https://pbs.twimg.com/media/C6OnJgRVUAExHzQ.jpg�https://twitter.com/eZ_XL1200C/status/838697771628544001/photo/1��rm���R��d�q�#
Objavro.schema�
{"type":"record","name":"Doc","doc":"adoc","fields":[
,
{"name":"user_friends_count","type":["int","null"]},
{"name":"user_location","type":["string","null"]},
{"name":"user_description","type":["string","null"]},
{"name":"user_statuses_count","type":["int","null"]},
{"name":"user_followers_count","type":["int","null"]},
{"name":"user_name","type":["string","null"]},
{"name":"user_screen_name","type":["string","null"]},
{"name":"created_at","type":["string","null"]},
{"name":"text","type":["string","null"]},
{"name":"retweet_count","type":["long","null"]},
{"name":"retweeted","type":["boolean","null"]},
{"name":"in_reply_to_user_id","type":["long","null"]},
{"name":"source","type":["string","null"]},
{"name":"in_reply_to_status_id","type":["long","null"]},
{"name":"media_url_https","type":["string","null"]},
{"name":"expanded_url","type":["string","null"]}]}I��
$
��b�u�St�XTޥ$838697775802040320��Exénekakavshi(2017-03-06T05:28:26Z�SUPER on est en alerte orange ils nous sortent quoi??? "faut pas s'approcher des toits les tuiles c fragile encore" mdrrrrr mrci fdp�<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>$838697775797792768� �full time working mom. convinced Runner. canadian at heart and #fantasydiamondrace Team holder. عبد الله الهرري��� running falahrunningfalah(2017-03-06T05:28:26Z�Heute läufts "wir brauchen dringendst Pasta zu Mittag" gut im Büro.�<a href="http://twitter.com" rel="nofollow">Twitter Web Client</a>$838697775818747904�|*دولة الكويت�من بغإني صرٺ #لعيونه ـ فدآ ومن تركنيء جعل ربيء يحفظه 💯 �m��Nضــّمـآ آلـمـشــاعـر
8vvvp(2017-03-06T05:28:26Z�RT @Sm_F__16: -