[SPARK-23878] unable to import col() or lit()

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.3.0
• Component/s: PySpark

Environment

eclipse 4.7.3

pyDev 6.3.2

pyspark==2.3.0

• Assignee: Unassigned
• Reporter: Andrew Davidson
• Created: 06/Apr/18 00:33
• Updated: 12/Dec/22 18:10
• Resolved: 06/Apr/18 16:44

Description

I have some code I am moving from a jupyter notebook to separate python modules. My notebook uses col() and list() and works fine

when I try to work with module files in my IDE I get the following errors. I am also not able to run my unit tests.

Description Resource Path Location Type
Unresolved import: lit load.py /adt_pyDevProj/src/automatedDataTranslation line 22 PyDev Problem

Description Resource Path Location Type
Unresolved import: col load.py /adt_pyDevProj/src/automatedDataTranslation line 21 PyDev Problem

I suspect that when you run pyspark it is generating the col and lit functions?

I found a discription of the problem @ https://stackoverflow.com/questions/40163106/cannot-find-col-function-in-pyspark I do not understand how to make this work in my IDE. I am not running pyspark just an editor

is there some sort of workaround or replacement for these missing functions?