[FLUME-1702] HDFSEventSink should write to a hidden file as opposed to a .tmp file

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Assignee: Jarek Jarcec Cecho
• Reporter: Brock Noland
• Created: 13/Nov/12 17:11
• Updated: 29/Aug/17 03:29
• Resolved: 18/Dec/12 20:05

Description

Currently we write to a .tmp file. The problem is that if MR jobs are being run on the directory we are writing to, then it's common for an MR job to list the directory, get a .tmp file and then in the mean time the .tmp file is renamed causing the job to fail when run.

Using JavaMR you can use a PathFilter to avoid this, however a custom solution is required for Pig, Hive, etc.

Perhaps we should write to a hidden file so that MR never tries to process data in flight.

Issue Links

• is duplicated by: FLUME-1704 HDFS sink: Add option to prefix .tmp files with some string like "_" or "."
• supercedes: FLUME-1486 Ability to configure a staging directory for data
• links to: Review board