[SPARK-23695] Confusing error message for PySpark's Kinesis tests when its jar is missing but enabled

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.1, 2.4.0
• Component/s: PySpark
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 15/Mar/18 12:37
• Updated: 12/Dec/22 17:51
• Resolved: 15/Mar/18 17:59

Description

Currently if its jar is missing but the Kinesis tests are enabled:

ENABLE_KINESIS_TESTS=1 SPARK_TESTING=1 bin/pyspark pyspark.streaming.tests
Skipped test_flume_stream (enable by setting environment variable ENABLE_FLUME_TESTS=1Skipped test_kafka_stream (enable by setting environment variable ENABLE_KAFKA_0_8_TESTS=1Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/.../spark/python/pyspark/streaming/tests.py", line 1572, in <module>
    % kinesis_asl_assembly_dir) +
NameError: name 'kinesis_asl_assembly_dir' is not defined

It shows a confusing error message. Seems a mistake.

Issue Links

• links to: [Github] Pull Request #20834 (HyukjinKwon)