Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3513

Provide a utility for running a function once on each executor

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • None
    • None
    • Spark Core
    • None

    Description

      This is minor, but it would be nice to have a utility where you can pass a function and it will run some arbitrary function once on each each executor and return the result to you (e.g. you could perform a jstack from within the JVM). You could probably hack it together with custom locality preferences, accessing the list of live executors, and mapPartitions.

      Attachments

        Issue Links

          duplicates

          New Feature - A new feature of the product, which has yet to be developed. SPARK-636 Add mechanism to run system management/configuration tasks on all workers

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-3514 Provide a utility function for returning the hosts (and number) of live executors

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              pwendell Patrick Wendell
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: