Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-2

Fine-grain control over stream consumption

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.7.0
    • Component/s: container
    • Labels:

      Description

      Currently, samza exposes configuration in the form of "streams.%s.consumer.max.bytes.per.sec" for throttling the # of bytes the Task will read from a stream. This is a feature request for programmatic fine-grain control over stream consumption. The use-case is a samza task that will be consuming multiple streams where some streams may be from live systems that have stricter SLA requirements and must always be prioritized over other streams that may be from batch systems. The above configuration is not the ideal way to express this type of stream prioritization because configuring the "batch" streams with a low consumption rate will decrease the overall throughput of the system when there is no data in the "live" streams. Furthermore, we'll want to throttle each "batch" stream based on external signals that can change over time. Because of the dynamic nature of these external signals, we would like to have a programmatic interface that can dynamically change the prioritization as the signal changes.

      Review board:

      https://reviews.apache.org/r/13725/

        Attachments

        1. SAMZA-2.0.patch
          23 kB
          Chris Riccomini
        2. SAMZA-2.1.patch
          28 kB
          Chris Riccomini
        3. SAMZA-2.2.patch
          99 kB
          Chris Riccomini
        4. SAMZA-2.3.patch
          92 kB
          Chris Riccomini
        5. SAMZA-2.4.patch
          97 kB
          Chris Riccomini
        6. SAMZA-2.5.patch
          120 kB
          Chris Riccomini
        7. DESIGN-SAMZA-2-0.pdf
          177 kB
          Chris Riccomini
        8. DESIGN-SAMZA-2-0.md
          22 kB
          Chris Riccomini

          Issue Links

            Activity

              People

              • Assignee:
                criccomini Chris Riccomini
                Reporter:
                criccomini Chris Riccomini
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: