[FLUME-1644] Shouldn't create 'log-xxx' file infinitely at space limitation local environment

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.2.0
• Component/s: Channel
• Assignee: Unassigned
• Reporter: Denny Ye
• Created: 16/Oct/12 08:55
• Updated: 16/Oct/12 15:28
• Resolved: 16/Oct/12 15:28

Description

There have consuming failure at FileChannel and Flume didn't delete inactive/consumed log file any more. Thus, 'No space left on device' was happening.

I want to raise another problem that in this case, there have too many log file with zero length. Flume holds so many file descriptors. Unluckily, Flume also creates log file every fixed minutes.

Look at my storage folder in local of Flume.

-rw------- 1 root root          0 Oct 16 00:26 log-6468.meta.tmp
-rw------- 1 root root          0 Oct 16 00:26 log-6469
-rw------- 1 root root          0 Oct 16 00:26 log-6469.meta.tmp
-rw------- 1 root root          0 Oct 16 00:26 log-6464
-rw------- 1 root root          0 Oct 16 00:26 log-6464.meta.tmp
-rw------- 1 root root          0 Oct 16 00:26 log-6465
-rw------- 1 root root          0 Oct 16 00:26 log-6465.meta.tmp
-rw------- 1 root root          0 Oct 16 00:26 log-6462
-rw------- 1 root root          0 Oct 16 00:26 log-6462.meta.tmp
-rw------- 1 root root          0 Oct 16 00:26 log-6463
-rw------- 1 root root          0 Oct 16 00:26 log-6463.meta.tmp
-rw------- 1 root root          0 Oct 16 00:26 log-6460
-rw------- 1 root root          0 Oct 16 00:26 log-6460.meta.tmp
-rw------- 1 root root          0 Oct 16 00:26 log-6461
-rw------- 1 root root          0 Oct 16 00:26 log-6461.meta.tmp
-rw------- 1 root root          0 Oct 16 00:25 log-6458
-rw------- 1 root root          0 Oct 16 00:25 log-6458.meta.tmp
-rw------- 1 root root          0 Oct 16 00:25 log-6459
-rw------- 1 root root          0 Oct 16 00:25 log-6459.meta.tmp
-rw------- 1 root root          0 Oct 16 00:25 log-6456
-rw------- 1 root root          0 Oct 16 00:25 log-6456.meta.tmp
-rw------- 1 root root          0 Oct 16 00:25 log-6457
-rw------- 1 root root          0 Oct 16 00:25 log-6457.meta.tmp
-rw------- 1 root root          0 Oct 16 00:25 log-6454.meta.tmp
-rw------- 1 root root          0 Oct 16 00:25 log-6455
...
...

It might be retry by SinkRunner when it will rollback in time, and invoke 'roll' method at every failure cases.

Issue Links

• duplicates: FLUME-1609 FileChannel detecting when the underlying file systems are full could provide cleaner error recovery