[SPARK-34033] SparkR Daemon Initialization

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: R, SparkR

Environment

tested on centos 7 & spark 2.3.1 and on my mac & spark at master

• Assignee: Unassigned
• Reporter: Tom Howland
• Due: 06/Jan/21
• Created: 06/Jan/21 20:16
• Updated: 04/Feb/21 18:41

Description

Provide a way for users to initialize the sparkR daemon before it forks.

I'm a contractor to Target, where we have several projects doing ML with sparkR. The changes proposed here result in weeks of compute-time saved with every run.

Please see docs/sparkr.md#daemon-initialization.

 

Issue Links

• links to: [Github] Pull Request #31162 (WamBamBoozle)