Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Done
    • None
    • SystemML 0.14
    • APIs, Runtime
    • None

    Description

      In contrast to parfor mr jobs, where every task has its own, process-local buffer pool, on spark with multi-threaded executors, multiple tasks share a common buffer pool. This is advantageous because common inputs are just read once. However, it also requires a synchronized buffer pool initialization and cleanup per executor. Especially the cleanup (e.g., of created cache directories) is tricky because spark does not provide an executor close call. Hence, our approach is to use a robust version of deleteOnExit that is independent of the exit code and also removes remaining files that are unknown during delete registration.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mboehm7 Matthias Boehm
            mboehm7 Matthias Boehm
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment