[SPARK-8489] Add regression tests for SPARK-8470

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.1, 1.5.0
• Component/s: SQL, Tests
• Labels: pull-request-available
• Target Version/s: 1.4.1, 1.5.0
• Assignee: Andrew Or
• Reporter: Andrew Or
• Created: 19/Jun/15 21:00
• Updated: 30/Sep/23 03:37
• Resolved: 20/Jun/15 00:37

Description

See SPARK-8470 for more detail. Basically the Spark Hive code silently overwrites the context class loader populated in SparkSubmit, resulting in certain classes missing when we do reflection in `SQLContext#createDataFrame`.

That issue is already resolved in https://github.com/apache/spark/pull/6891, but we should add a regression test for the specific manifestation of the bug in SPARK-8470.

Issue Links

• is related to: SPARK-8470 MissingRequirementError for ScalaReflection on user classes
• links to: [Github] Pull Request #11630 (dongjoon-hyun)
• links to: [Github] Pull Request #11744 (dongjoon-hyun)
• links to: [Github] Pull Request #12924 (dilipbiswal)
• links to: [Github] Pull Request #22308 (sadhen)
• links to: GitHub Pull Request #43179