XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: SystemML 0.14
    • Component/s: APIs, Runtime
    • Labels:
      None

      Description

      This task aims to make the memory handling of re-used and temporary broadcast variables more robust in order to avoid unnecessary OOMs.

      1) Explicitly destroy temporary broadcast variables because we would otherwise never clean them up and Spark's ContextCleaner seems to be a best effort daemon that triggers every 30min or on garbage collection (which might be too late if a large object is allocated)

      2) Keep track of currently softly reachable (re-used) broadcasts in order to take their size into account when deciding on guarded collect.

        Attachments

          Activity

            People

            • Assignee:
              mboehm7 Matthias Boehm
              Reporter:
              mboehm7 Matthias Boehm
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: