Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35498

Add an API "inheritable_thread_target" which return a wrapped thread target for pyspark pin thread mode

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • PySpark
    • None

    Description

      In pyspark, user may create some threads, not via `Thread` object, but via some parallel helper function such as:
      `thread_pool.imap_unordered`

      In this case, we need create a function wrapper, used in pin thread mode, The wrapper function, before calling original thread target, it inherits the inheritable properties specific to JVM thread such as ``InheritableThreadLocal``, and after original thread target return, garbage-collects the Python thread instance and also closes the connection which finishes JVM thread correctly.

      Attachments

        Activity

          People

            weichenxu123 Weichen Xu
            weichenxu123 Weichen Xu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: