-
Type:
Bug
-
Status: Open
-
Priority:
Critical
-
Resolution: Unresolved
-
Affects Version/s: 1.2.2
-
Fix Version/s: None
-
Component/s: storm-hdfs
-
Labels:None
We just discovered when using a SequenceFileBolt (although it might happen with other implementations as well) that the writers it uses, held in the map AbstractHdfsBolt.writers are not closed/synced when they are removed from the map by the removeEldestEntry method.
This leads to data loss.
Can be reproduced by creating a SequenceFileBolt.withMaxOpenFiles(1) and writing to just two different files.
One will have a size of zero, the other has the data in it.