Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22376

run-tests.py fails at exec-sbt if run with Python 3

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.0
    • Component/s: Tests
    • Labels:
      None
    • Environment:

      OSX 10.12.6 Python 3.6.0 Anaconda 4.4

      Description

      Running ./dev/run-tests with python3 as the default gives this error at the Building Spark stage:

      ========================================================================
      Building Spark
      ========================================================================
      [info] Building Spark (w/Hive 1.2.1) using SBT with these arguments:  -Phadoop-2.6 -Phive-thriftserver -Phive -Pkafka-0-8 -Pflume -Pyarn -Pmesos -Pkinesis-asl test:package streaming-kafka-0-8-assembly/assembly streaming-flume-assembly/assembly streaming-kinesis-asl-assembly/assembly
      Traceback (most recent call last):
        File "./dev/run-tests.py", line 622, in <module>
          main()
        File "./dev/run-tests.py", line 593, in main
          build_apache_spark(build_tool, hadoop_version)
        File "./dev/run-tests.py", line 391, in build_apache_spark
          build_spark_sbt(hadoop_version)
        File "./dev/run-tests.py", line 344, in build_spark_sbt
          exec_sbt(profiles_and_goals)
        File "./dev/run-tests.py", line 293, in exec_sbt
          if not sbt_output_filter.match(line):
      TypeError: cannot use a string pattern on a bytes-like object
      

      This is because in Python 3, the stdout member of a POpen object defaults to returning a byte stream, and exec_sbt tries to read it as a text screen. This can be fixed by specifying universal_newlines=True when creating the POpen object. I notice that the hashbang at the start of run-tests.py says to run with python2, so I am not sure how much of the rest of it is Python 3 compatible, but this is the first error I've run into. It ran with python 3 because run-tests runs run-tests.py using the default Python, which on my system is Python 3. Not sure whether the better solution here is to try and fix Python 3 compatibility in run-tests.py or set run-tests to use Python 2.

        Attachments

          Activity

            People

            • Assignee:
              hyukjin.kwon Hyukjin Kwon
              Reporter:
              TV4Fun Joel Croteau
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: