Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29396

Extend Spark plugin interface to driver

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core
    • Labels:

      Description

      Spark provides an extension API for people to implement executor plugins, added in SPARK-24918 and later extended in SPARK-28091.

      That API does not offer any functionality for doing similar things on the driver side, though. As a consequence of that, there is not a good way for the executor plugins to get information or communicate in any way with the Spark driver.

      I've been playing with such an improved API for developing some new functionality. I'll file a few child bugs for the work to get the changes in.

        Attachments

          Activity

            People

            • Assignee:
              vanzin Marcelo Masiero Vanzin
              Reporter:
              vanzin Marcelo Masiero Vanzin
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: