[FLUME-3233] log roll to a new name,the flume collect repeat

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.7.0
• Assignee: Unassigned
• Reporter: 67287685
• Created: 03/May/18 06:10
• Updated: 22/Dec/20 07:30

Description

we use the flume 1.7 to collect our log send to kafka. 

our java application  compressed the log at every day 00:00,the log name change aaa.log -> aaa.20180503.tmp -> aaa.20180503.log.gz,the change while complete less than 1 second time.

this have a problem ,when have some data into log at 23:59:59,the log while compress fast,the flume not collect the new data,the log is become zip file,so some data will lost.

To solve this problem，we Modify the java application and flume

1.modify java application

the log is compress delay 5 minutes，aaa.log --> aaa.20180503.tmp -> after 5 min -> aaa.20180503.log.gz

the flume have enough time complete collect the data .

 

2.modify flume 1.7

use inode judge the file uniqueness，not use inode + filename ,when use inode + filename，the flume while repeate collect when the aaa.log change to aaa.20180503.tmp.this two file have the same inode,we will not repeat collect the file.

 

Please help to see if our plan is feasible，thanks every .