[SPARK-37620] Use more precise types for SparkContext Optional fields (i.e. _gateway, _jvm)

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: PySpark
• Assignee: Byron Hsu
• Reporter: Maciej Szymkiewicz
• Created: 12/Dec/21 21:45
• Updated: 18/Dec/21 10:27
• Resolved: 17/Dec/21 19:53

Description

As a part of SPARK-37152 we agreed to keep these typed as not Optional for simplicity, but this just a temporary solution to move things forward, until we decide on best approach to handle such cases.

Issue Links

• links to: [Github] Pull Request #34892 (ByronHsu)
• links to: [Github] Pull Request #34892 (ByronHsu)
• links to: [Github] Pull Request #34944 (zero323)