Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2557

A bug in DisruptorQueue causing severe underestimation of queue arrival rates

    Details

      Description

      Recently, we are tuning the performance of our topology and deploying some theoretical performance models that heavily rely on the metrics of query arrival rates. We found a bug in DisruptorQueue that leads to severe underestimation to the queue arrival rates. After further investigation, we finally found that in the current implementation of DisruptorQueue, the arrival rates are actually measured as the number of batches of tuples rather than the actual number of tuples, resulting in significant underestimation of the arrival rates.

      To be more specific, in DisruptorQueue.publishDirectSingle() and DisruptorQueue.publishDirect() functions, objects containing tuples are published to the buffer and the metrics are notified by calling _metric.notifyArrivals(1). This works fine when the object is simply a wrapper of a single tuple. However, the object could also be an instance of ArrayList<AddressedTuple> or HashMap<Integer, ArrayList<TaskMessage>>. In such case, we should get the actual number of tuples in the object and notify the _metrics with the right value.

        Attachments

          Activity

            People

            • Assignee:
              wendyshusband tangkailin
              Reporter:
              wendyshusband tangkailin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 3h 10m
                3h 10m