Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23695

Confusing error message for PySpark's Kinesis tests when its jar is missing but enabled

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Trivial
    • Resolution: Fixed
    • 2.3.0
    • 2.3.1, 2.4.0
    • PySpark
    • None

    Description

      Currently if its jar is missing but the Kinesis tests are enabled:

      ENABLE_KINESIS_TESTS=1 SPARK_TESTING=1 bin/pyspark pyspark.streaming.tests
      Skipped test_flume_stream (enable by setting environment variable ENABLE_FLUME_TESTS=1Skipped test_kafka_stream (enable by setting environment variable ENABLE_KAFKA_0_8_TESTS=1Traceback (most recent call last):
        File "/usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
          "__main__", fname, loader, pkg_name)
        File "/usr/local/Cellar/python/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
          exec code in run_globals
        File "/.../spark/python/pyspark/streaming/tests.py", line 1572, in <module>
          % kinesis_asl_assembly_dir) +
      NameError: name 'kinesis_asl_assembly_dir' is not defined
      

      It shows a confusing error message. Seems a mistake.

      Attachments

        Activity

          People

            hyukjin.kwon Hyukjin Kwon
            hyukjin.kwon Hyukjin Kwon
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: