Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-1308 Runtime feature extensions
  3. SYSTEMDS-1879

Parfor remote spark w/ reuse of shared inputs

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: SystemML 0.15
    • Component/s: APIs, Runtime
    • Labels:
      None

      Description

      Currently, we read shared inputs redundantly in each parfor worker. This causes redundant read and is unnecessarily memory-inefficient.

      This task aims to read shared inputs once per process and reuse them across threads. The most elegant way of handling this is to reuse initially parsed symbol table entries (instances of matrix objects), except for result variables. Then the sharing happens automatically (similar to local parfor) over the shared per-process buffer pool.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mboehm7 Matthias Boehm
                Reporter:
                mboehm7 Matthias Boehm
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: