[SPARK-25236] Investigate using a logging library inside of PySpark on the workers instead of print

• Type: Improvement
• Status: Open
• Priority: Trivial
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Holden Karau
• Shepherd: Holden Karau
• Created: 24/Aug/18 21:25
• Updated: 16/Mar/20 22:50

Description

We don't have a logging library on the workers to use which means that its difficult for folks to tune the log level on the workers. On the driver processes we could just call the JVM logging, but on the workers that won't work.