Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29780

The UI can access into the ResourceAllocator, whose data structures are being updated from scheduler threads

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.0.0
    • None
    • Spark Core
    • None

    Description

      A class extending ResourceAllocator (WorkerResourceInfo), has some potential issues (raised here: https://github.com/apache/spark/pull/26078#discussion_r340342820)

      The WorkerInfo class is calling into availableAddrs and assignedAddrs but those calls appear to be coming from the UI (looking at the resourcesInfo* functions), e.g. JsonProtocol and MasterPage call this. Since the datastructures in ResourceAllocator are not concurrent, we could end up with bad data or potentially crashes, depending on when the calls are made.

      Note that there are other calls to the resourceInfo* functions, but those are from the event loop.

      Attachments

        Issue Links

          relates to

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SPARK-24615 SPIP: Accelerator-aware task scheduling for Spark

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              abellina Alessandro Bellina
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: