Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12753

Import error during unit test while calling a function from reduceByKey()

    XMLWordPrintableJSON

Details

    • Question
    • Status: Closed
    • Trivial
    • Resolution: Not A Problem
    • 1.6.0
    • None
    • PySpark
    • El Capitan, Single cluster Hadoop, Python 3, Spark 1.6, Anaconda

    Description

      The current directory structure for my test script is as follows:
      project/
      script/
      _init_.py
      map.py
      test/
      _init.py_
      test_map.py

      I have attached map.py and test_map.py file with this issue.

      When I run the nosetest in the test directory, the test fails. I get no module named "script" found error.
      However when I modify the map_add function to replace the call to add within reduceByKey in map.py like this:

      def map_add(df):
      result = df.map(lambda x: (x.key, x.value)).reduceByKey(lambda x,y: x+y)
      return result

      The test passes.

      Also, when I run the original test_map.py from the project directory, the test passes.

      I am not able to figure out why the test doesn't detect the script module when it is within the test directory.

      I have also attached the log error file. Any help will be much appreciated.

      Attachments

        1. test_map.py
          1 kB
          Dat Tran
        2. map.py
          0.1 kB
          Dat Tran
        3. log.txt
          67 kB
          Dat Tran

        Activity

          People

            Unassigned Unassigned
            datitran Dat Tran
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: