[ZEPPELIN-4802] pyspark warnings with spark-3.0.0-preview - Sync of properties between JVM and PVM

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.0
• Assignee: Jeff Zhang
• Reporter: Philipp Dallig
• Created: 07/May/20 12:07
• Updated: 24/Dec/20 03:14
• Resolved: 01/Jun/20 08:45

Description

Hi

I'm using Zeppelin (Version: master branch) with spark-3.0.0-preview. I get the following warning if I start a simple pyspark paragraph. Can someone help to solve this issue?

pyspark code

%pyspark

spark

Warning

/opt/spark/python/lib/pyspark.zip/pyspark/util.py:141: UserWarning: Currently, 'setJobGroup' (set to local properties) with multiple threads does not properly work. 
Internally threads on PVM and JVM are not synced, and JVM thread can be reused for multiple threads on PVM, which fails to isolate local properties for each thread on PVM. 
To work around this, you can set PYSPARK_PIN_THREAD to true (see SPARK-22340). However, note that it cannot inherit the local properties from the parent thread although it isolates each thread on PVM and JVM with its own local properties. 
To work around this, you should manually copy and set the local properties from the parent thread to the child thread when you create another thread.
/opt/spark/python/lib/pyspark.zip/pyspark/util.py:141: UserWarning: Currently, 'setLocalProperty' (set to local properties) with multiple threads does not properly work. 
Internally threads on PVM and JVM are not synced, and JVM thread can be reused for multiple threads on PVM, which fails to isolate local properties for each thread on PVM. 
To work around this, you can set PYSPARK_PIN_THREAD to true (see SPARK-22340). However, note that it cannot inherit the local properties from the parent thread although it isolates each thread on PVM and JVM with its own local properties. 
To work around this, you should manually copy and set the local properties from the parent thread to the child thread when you create another thread.