Minor Description
Although you can make pyspark use ipython with IPYTHON=1, and also change the python executable with PYSPARK_PYTHON=..., you can't use both at the same time because it hardcodes the default ipython script.
This makes it use the PYSPARK_PYTHON variable if present and fall back to default python, similarly to how the default python executable is handled.
So you can use a custom ipython like so:
PYSPARK_PYTHON=./anaconda/bin/ipython IPYTHON_OPTS="notebook" pyspark