Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12753

Import error during unit test while calling a function from reduceByKey()

    Details

    • Type: Question
    • Status: Closed
    • Priority: Trivial
    • Resolution: Not A Problem
    • Affects Version/s: 1.6.0
    • Fix Version/s: None
    • Component/s: PySpark
    • Environment:

      El Capitan, Single cluster Hadoop, Python 3, Spark 1.6, Anaconda

      Description

      The current directory structure for my test script is as follows:
      project/
      script/
      _init_.py
      map.py
      test/
      _init.py_
      test_map.py

      I have attached map.py and test_map.py file with this issue.

      When I run the nosetest in the test directory, the test fails. I get no module named "script" found error.
      However when I modify the map_add function to replace the call to add within reduceByKey in map.py like this:

      def map_add(df):
      result = df.map(lambda x: (x.key, x.value)).reduceByKey(lambda x,y: x+y)
      return result

      The test passes.

      Also, when I run the original test_map.py from the project directory, the test passes.

      I am not able to figure out why the test doesn't detect the script module when it is within the test directory.

      I have also attached the log error file. Any help will be much appreciated.

        Attachments

        1. map.py
          0.1 kB
          Dat Tran
        2. test_map.py
          1 kB
          Dat Tran
        3. log.txt
          67 kB
          Dat Tran

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              datitran Dat Tran
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: