[STORM-3698] AbstractHdfsBolt does not sync Writers that are purged

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.2.2
• Component/s: storm-hdfs
• Assignee: Unassigned
• Reporter: Max Schmidt
• Created: 11/Sep/20 09:27
• Updated: 11/Sep/20 09:27

Description

We just discovered when using a SequenceFileBolt (although it might happen with other implementations as well) that the writers it uses, held in the map AbstractHdfsBolt.writers are not closed/synced when they are removed from the map by the removeEldestEntry method.

This leads to data loss.

Can be reproduced by creating a SequenceFileBolt.withMaxOpenFiles(1) and writing to just two different files.

One will have a size of zero, the other has the data in it.