Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-495

Add delayed retries to KafkaSpout

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.9.3
    • Fix Version/s: 0.10.0
    • Component/s: storm-kafka
    • Labels:
    • Environment:
      all environments

      Description

      If a tuple in the topology originates from the KafkaSpout from the external/storm-kafka sources, and if a bolt in the topology indicates a failure by calling fail() on its OutputCollector, the KafkaSpout will immediately retry the message.

      We wish to use this failure and retry behavior in our ingestion system whenever we experience a recoverable error from a downstream system, such as a 500 or 503 error from a service we depend on. But with the current KafkaSpout behavior, doing so results in a tight loop where we retry several times over a few seconds and then give up. I want to be able to delay retry to give the downstream service some time to recover. Ideally, I would like to have configurable, exponential backoff retry.

        Attachments

          Activity

            People

            • Assignee:
              rkilgore Rick Kilgore
              Reporter:
              rkilgore Rick Kilgore
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: