[FLUME-518] Flume is consistently using 70%+ of the CPU

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 0.9.1
• Component/s: Node
• Labels: performance tailDir

Environment

Amazon EC2 Large instance running CentOS5.4: Linux domU-12-31-39-01-F1-31 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

• Assignee: Jonathan Hsieh
• Reporter: Disabled imported user
• Created: 11/Feb/11 23:43
• Updated: 15/Mar/11 14:52
• Resolved: 15/Mar/11 14:51

Description

With a fairly light output of ~1-2M per second, Flume typically sits at around 75-85% of the CPU (see below). I am currently using tailDir and it is tailing 12 files. The vast majority of the 1-2M is coming in on 4 of those file. Another 4 see probably about 3-5k/s and the 3rd set of 4 is somewhere in between. Also spikes at the speed at which data comes in sends the CPU through the roof (I'm assuming while Flume tries to keep up).

2 potential solutions:
1 (from Eric Sammer) Make the file polling frequency configurable (not hard coded) to allow for less frequent checks.
2 (from Jon Hsieh) Back off after 2 reads with no data

– Top output –
top - 23:26:22 up 11 days, 7:24, 3 users, load average: 1.02, 0.91, 0.84
Tasks: 67 total, 2 running, 65 sleeping, 0 stopped, 0 zombie
Cpu(s): 17.1%us, 27.1%sy, 0.0%ni, 36.8%id, 0.0%wa, 0.0%hi, 0.2%si, 18.9%st
Mem: 7872040k total, 7851560k used, 20480k free, 26800k buffers
Swap: 0k total, 0k used, 0k free, 6023020k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5151 flume 19 0 2198m 235m 4492 S 75.0 3.1 783:53.90 java
22897 ad 15 0 867m 246m 6140 S 4.0 3.2 7:46.20 node
22898 ad 16 0 867m 246m 6132 S 4.0 3.2 7:46.09 node
22896 ad 15 0 868m 248m 6132 S 3.7 3.2 7:39.38 node
22900 ad 15 0 868m 249m 6132 S 1.3 3.2 7:35.49 node
9889 nobody 15 0 40032 2092 708 S 0.7 0.0 15:11.97 nginx
9891 nobody 15 0 40108 2248 700 S 0.3 0.0 15:06.09 nginx
1 root 15 0 10348 756 632 S 0.0 0.0 0:00.54 init

– Free output
[root@domU-12-31-39-01-F1-31 ~]# free -m
total used free shared buffers cached
Mem: 7687 7664 23 0 25 5880
-/+ buffers/cache: 1757 5929
Swap: 0 0 0

– vmstat 1
procs ----------memory--------- --swap- ----io--- -system- ----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 15212 26168 6020636 0 0 97 179 5 13 12 7 66 6 9
3 0 0 26064 26168 6021256 0 0 504 0 2418 3093 24 26 31 0 20
1 0 0 20728 26168 6021944 0 0 500 4 2852 3314 14 27 42 0 17
1 0 0 22396 26180 6022520 0 0 500 928 2584 3042 20 24 36 0 20
1 0 0 18452 26180 6023208 0 0 504 0 2770 3378 16 26 40 0 18
1 0 0 14068 26180 6023920 0 0 508 0 2629 3027 14 28 41 0 16
2 0 0 26596 26180 6021512 0 0 500 0 2601 3042 26 25 32 0 18

– iostat 1
avg-cpu: %user %nice %system %iowait %steal %idle
16.50 0.00 24.50 0.00 18.00 41.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda1 0.00 0.00 0.00 0 0
sdb 14.00 1008.00 0.00 1008 0
sdc 0.00 0.00 0.00 0 0

avg-cpu: %user %nice %system %iowait %steal %idle
20.10 0.00 23.62 0.50 20.10 35.68
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda1 0.00 0.00 0.00 0 0
sdb 70.00 1008.00 2000.00 1008 2000
sdc 0.00 0.00 0.00 0 0

Issue Links

• duplicates: FLUME-252 Update Tail to get rid of races and truncation problems.