Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25181

Block Manager master and slave thread pools are unbounded

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.4.0
    • Component/s: Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      Currently, BlockManagerMasterEndpoint and BlockManagerSlaveEndpoint both have thread pools with unbounded numbers of threads. In certain cases, this can lead to driver OOM errors. We should add an upper bound on the number of threads in these thread pools; this should not break any existing behavior because they still have queues of size Integer.MAX_VALUE.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mukulmurthy Mukul Murthy
                Reporter:
                mukulmurthy Mukul Murthy
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: