[SPARK-12753] Import error during unit test while calling a function from reduceByKey()

• Type: Question
• Status: Closed
• Priority: Trivial
• Resolution: Not A Problem
• Affects Version/s: 1.6.0
• Component/s: PySpark
• Labels: pyspark python3 unit-test

Environment

El Capitan, Single cluster Hadoop, Python 3, Spark 1.6, Anaconda

• External issue URL: http://stackoverflow.com/questions/34724215/import-error-during-unit-test-while-calling-a-function-from-reducebykey
• Assignee: Unassigned
• Reporter: Dat Tran
• Created: 11/Jan/16 14:54
• Updated: 16/Oct/16 13:01
• Resolved: 16/Oct/16 12:58

Description

The current directory structure for my test script is as follows:
project/
script/
_init_.py
map.py
test/
_init.py_
test_map.py

I have attached map.py and test_map.py file with this issue.

When I run the nosetest in the test directory, the test fails. I get no module named "script" found error.
However when I modify the map_add function to replace the call to add within reduceByKey in map.py like this:

def map_add(df):
result = df.map(lambda x: (x.key, x.value)).reduceByKey(lambda x,y: x+y)
return result

The test passes.

Also, when I run the original test_map.py from the project directory, the test passes.

I am not able to figure out why the test doesn't detect the script module when it is within the test directory.

I have also attached the log error file. Any help will be much appreciated.