[SPARK-5063] Display more helpful error messages for several invalid operations

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.1, 1.3.0
• Component/s: Spark Core
• Target Version/s: 1.2.1
• Assignee: Josh Rosen Assign to me
• Reporter: Josh Rosen
• Created: 03/Jan/15 03:34
• Updated: 12/Aug/19 23:45
• Resolved: 24/Jan/15 01:54

Description

Spark does not support nested RDDs or performing Spark actions inside of transformations; this usually leads to NullPointerExceptions (see SPARK-718 as one example). The confusing NPE is one of the most common sources of Spark questions on StackOverflow:

• https://stackoverflow.com/questions/13770218/call-of-distinct-and-map-together-throws-npe-in-spark-library/14130534#14130534
• https://stackoverflow.com/questions/23793117/nullpointerexception-in-scala-spark-appears-to-be-caused-be-collection-type/23793399#23793399
• https://stackoverflow.com/questions/25997558/graphx-ive-got-nullpointerexception-inside-mapvertices/26003674#26003674

(those are just a sample of the ones that I've answered personally; there are many others).

I think we can detect these errors by adding logic to RDD to check whether sc is null (e.g. turn sc into a getter function); we can use this to add a better error message.

In PySpark, these errors manifest themselves slightly differently. Attempting to nest RDDs or perform actions inside of transformations results in pickle-time errors:

rdd1 = sc.parallelize(range(100))
rdd2 = sc.parallelize(range(100))
rdd1.mapPartitions(lambda x: [rdd2.map(lambda x: x)])

produces

[...]
  File "/Users/joshrosen/anaconda/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/Users/joshrosen/Documents/Spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
  File "/Users/joshrosen/Documents/Spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 304, in get_return_value
py4j.protocol.Py4JError: An error occurred while calling o21.__getnewargs__. Trace:
py4j.Py4JException: Method __getnewargs__([]) does not exist
	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:333)
	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:342)
	at py4j.Gateway.invoke(Gateway.java:252)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:207)
	at java.lang.Thread.run(Thread.java:745)

We get the same error when attempting to broadcast an RDD in PySpark. For Python, improved error reporting could be as simple as overriding the getnewargs method to throw a more useful UnsupportedOperation exception with a more helpful error message.

Users may also see confusing NPEs when calling methods on stopped SparkContexts, so I've added checks for that as well.

Issue Links

• is duplicated by: SPARK-15787 Display more helpful error messages for several invalid operations
• is related to: SPARK-5065 BroadCast can still work after sc had been stopped.
• is related to: SPARK-28702 Display useful error message (instead of NPE) for invalid Dataset operations (e.g. calling actions inside of transformations)
• links to: [Github] Pull Request #3884 (JoshRosen)