[FLUME-2113] Flume: No errors thrown but flume fails to transfer the file completely

• Type: Question
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Labels: hdfs sink
• Assignee: Unassigned
• Reporter: kasa v varun
• Created: 26/Jun/13 08:01
• Updated: 26/Jun/13 08:16

Description

Flow: using a basic flow, spoolDir -> fileChannel -> HDFS

No extra parameters set in the .conf file

File size that i'm trying to transfer: 1.4GB

Situation: the agents starts fine, the file transfer starts fine, the file in source gets renamed to .COMPLETED, the complete file is not getting transfered to HDFS, no errors/exceptions are being thrown. I ran the same adhoc several times i found that out of the 1.4 gigs only ~169Mb is getting transfered. Seems weird !

Any suggestions? Any solutions? any hypotheses?