Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-775

Specify prefetch buffer in terms of bytes rather than count using fetchThreshold

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0, 0.9.1
    • Fix Version/s: 0.10.0, 0.9.1
    • Component/s: None
    • Labels:
      None

      Description

      Currently the only way to control the prefetch buffer is count based using the systems.system-name.samza.fetch.threshold configuration. However, in the presence of variable sized messages this make it very hard to allocate determisnistic memory resources to a SamzaContainer.

      This JIRA is a for an improvment to allow tuning the prefetch buffer based on bytes as well using a new config like systems.source.samza.fetch.bytes. When this is present the threshold could be safely ignored.

      This is an extremely important feature for us as it allowed us to stabilize our platform as we have deterministic allocations.

      I have a patch of this against 0.9.1 running in canaries at scale in prod and is looking promising so far.

        Attachments

        1. samza-775.patch
          28 kB
          Monal Daxini
        2. samza-775-design-doc.md
          4 kB
          Monal Daxini
        3. samza-775-design-doc.pdf
          39 kB
          Monal Daxini

          Issue Links

            Activity

              People

              • Assignee:
                mdaxini Monal Daxini
                Reporter:
                mdaxini Monal Daxini
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: