Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3626

Replace AsyncRDDActions with a more general async. API

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      The experimental AsyncRDDActions APIs seem to only exist in order to enable job cancellation.

      We've been considering extending these APIs to support progress monitoring, but this would require stabilizing them so they're no longer @Experimental.

      Instead, I propose to replace all of the AsyncRDDActions with a mechanism based on job groups which allows arbitrary computations to be run in job groups and supports cancellation / monitoring of Spark jobs launched from those computations.

      (full design pending; see my GitHub PR for more details).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                joshrosen Josh Rosen
                Reporter:
                joshrosen Josh Rosen
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: