[SPARK-26080] Unable to run worker.py on Windows

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.1, 3.0.0
• Component/s: PySpark
• Labels: release-notes

Environment

Windows 10 Education 64 bit

• Target Version/s: 2.4.1, 3.0.0
• Assignee: Hyukjin Kwon
• Reporter: Hayden Jeune
• Created: 15/Nov/18 21:03
• Updated: 12/Dec/22 18:10
• Resolved: 02/Dec/18 09:41

Description

Use of the resource module in python means worker.py cannot run on a windows system. This package is only available in unix based environments.
https://github.com/apache/spark/blob/9a5fda60e532dc7203d21d5fbe385cd561906ccb/python/pyspark/worker.py#L25

textFile = sc.textFile("README.md")
textFile.first()

When the above commands are run I receive the error 'worker failed to connect back', and I can see an exception in the console coming from worker.py saying 'ModuleNotFoundError: No module named resource'

I do not really know enough about what I'm doing to fix this myself. Apologies if there's something simple I'm missing here.

Issue Links

• is duplicated by: SPARK-26670 worker.py uses resource python library, which is not available for windows
• relates to: SPARK-25004 Add spark.executor.pyspark.memory config to set resource.RLIMIT_AS
• links to: [Github] Pull Request #23055 (HyukjinKwon)
• links to: GitHub Pull Request #23055