Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2423

Join Bolt : Use explicit instead of default window anchoring of emitted tuples

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0, 1.x
    • Fix Version/s: 2.0.0, 1.1.0
    • Component/s: storm-core
    • Labels:
      None

      Description

      Default anchoring will anchor each emitted tuple to every tuple in current window. This requires a very large numbers of ACKs from any downstream bolt. If topology.debug is enabled, it also worsens the load on the system significantly.

      Letting the topo run in this mode (in particular with max.spout.pending disabled), could lead to the worker running out of memory and crashing.

      Fix: Join Bolt should avoid using default window anchoring, and explicitly anchor each emitted tuple with the exact matching tuples form each inputs streams. This reduces the complexity of the tuple trees and consequently the reduces burden on the ACKing & messaging subsystems.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                roshan_naik Roshan Naik
                Reporter:
                roshan_naik Roshan Naik
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 50m
                  1h 50m