Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-1591

Planner for high-level applications should validate the partition number of intermediate topics

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.15.0
    • Component/s: None
    • Labels:

      Description

      We have seen cases where the intermediate topic created by partitionBy() operator was created with partition number calculated by the planner. However, when input topic partition expands with traffic, usually that would have trickling effect to the downstream topic partition and task assignments (i.e. 128 partitions from the input topic got shuffled into only 8 partitions in the intermediate topic usually does not distribute the work load widely enough). At least, the planner needs to validation and print a warning message if the intermediate topic partition is less than the max of input/output topic partition number.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              nickpan47 Yi Pan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: