Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-2044

EOSMessage causes out of memory Exceptions related to WindowOperatorImpl

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.14.0, 1.0
    • Fix Version/s: None
    • Component/s: container
    • Labels:
      None

      Description

      The contract of the handleEndOfStream method dictates that a collection of results be returned. In the case of WindowOperatorImpl which has a backing RocksDB store, this effectively causes the entirety of the store to be pulled into memory. In many cases, this will cause out of memory exceptions (otherwise why not keep the store in memory in the first place).

      Since this is a protected api, I have a relatively simple change to propose which could allow data to be consumed downstream as it is brought out of the store:

      protected void handleEndOfStream(Consumer<WindowPane<K, Object>> consumer, MessageCollector collector, TaskCoordinator coordinator)

      Pull Request 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                pgeorgantas Peter Georgantas
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: