Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3698

AbstractHdfsBolt does not sync Writers that are purged

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.2.2
    • Fix Version/s: None
    • Component/s: storm-hdfs
    • Labels:
      None

      Description

      We just discovered when using a SequenceFileBolt (although it might happen with other implementations as well) that the writers it uses, held in the map AbstractHdfsBolt.writers are not closed/synced when they are removed from the map by the removeEldestEntry method.

      This leads to data loss.

      Can be reproduced by creating a SequenceFileBolt.withMaxOpenFiles(1) and writing to just two different files.

      One will have a size of zero, the other has the data in it. 

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jondoe1337 Max Schmidt
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: