[FLUME-2710] Timestamp file extension

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.6.0, 1.5.1
• Component/s: Configuration, Sinks+Sources
• Assignee: Unassigned
• Reporter: Maciej Karaś
• Created: 28/May/15 09:47
• Updated: 19/Sep/15 17:52

Description

Timestamp extension is added to every file in HDFS sink. For a matter, it is added to every sink using BucketWriter.java. Line 247 -249 adds this timestamp counter to filename and it cannot be configured in opposite to what ashar103 said in parent task.

        long counter = fileExtensionCounter.incrementAndGet();

        String fullFileName = fileName + "." + counter;
[...]
        bucketPath = filePath + "/" + inUsePrefix + fullFileName + inUseSuffix;
        targetPath = filePath + "/" + fullFileName;

Event creation time is not necessary a time of event saving into hdfs, obviously events can be delayed. Thus I don't know why file creation timestamp is mandatory - it can be confusing.

Issue Links

• incorporates: FLUME-2703 HDFS sink: Ability to exclude time counter in fileName via sink configuration