[HADOOP-12759] RollingFileSystemSink should eagerly rotate directories

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Target Version/s: 2.9.0
• Assignee: Daniel Templeton
• Reporter: Daniel Templeton
• Created: 02/Feb/16 14:20
• Updated: 30/Aug/16 01:19
• Resolved: 07/Feb/16 04:55

Description

The RollingFileSystemSink only rolls over to a new directory if a new metrics record comes in. The issue is that HDFS does not update the file size until it's closed (HDFS-5478), and if no new metrics record comes in, then the file size will never be updated.

This JIRA is to add a background thread to the sink that will eagerly close the file at the top of the hour.

Issue Links

• is depended upon by: HDFS-9780 RollingFileSystemSink doesn't work on secure clusters
• is depended upon by: HDFS-9637 Tests for RollingFileSystemSink