Major Description
For Python UDF, we can support runtime configuration to set Python workers. It will create a new Python worker if the configuration is changed.
This is especially useful when you want to run Spark with different dependencies during runtime. See also https://www.databricks.com/blog/2020/12/22/how-to-manage-python-dependencies-in-pyspark.html