[SAMZA-744] Samza shutdown sequence throws exception

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0, 0.9.1
• Fix Version/s: 0.10.0
• Assignee: Yi Pan
• Reporter: Yi Pan
• Created: 28/Jul/15 23:51
• Updated: 08/Aug/15 00:06
• Resolved: 08/Aug/15 00:06

Description

The current shutdown sequence has a problematic sequence for changelog attached stores:

    } finally {
      info("Shutting down.")

      shutdownConsumers
      shutdownTask
      shutdownProducers
      shutdownStores
      shutdownLocalityManager
      shutdownOffsetManager
      shutdownMetrics

      info("Shutdown complete.")
    }

If the KV-stores has pending writes in cache and a changelog attached, shutdownStores will try to flush the store, which results in invoking producer.send(). However, the producers are already shutdown according to the above sequence. We need to move the shutdownProducers later than any shutdown procedure that would possibly flushing out pending writes to the system streams.