[SPARK-7006] Inconsistent behavior for ctrl-c in Spark shells

• Type: Wish
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.3.1
• Component/s: Spark Shell, YARN
• Labels: shell yarn

Environment

YARN

• Assignee: Unassigned
• Reporter: Cheolsoo Park
• Created: 20/Apr/15 07:01
• Updated: 18/Feb/16 10:06
• Resolved: 18/Feb/16 10:06

Description

When ctrl-c is pressed in shell, behaviors are not consistent across spark-sql, spark-shell, and pyspark resulting in confusion for users. Here is the summary-

shell	after ctrl-c
spark-sql	cancels the running job
spark-shell	exits the shell
pyspark	throws error [1] and doesn't cancel the job

Particularly, pyspark is worst because it gives a wrong impression that the job is cancelled although it is not.

Ideally, every shell should act like spark-sql because it allows users to cancel the running job while staying in shell. (Pressing ctrl-c twice exits the shell.)

[1] pyspark error for ctrl-c

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/cheolsoop/spark/jars/spark-1.3.1/python/pyspark/sql/dataframe.py", line 284, in count
    return self._jdf.count()
  File "/home/cheolsoop/spark/jars/spark-1.3.1/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 536, in __call__
  File "/home/cheolsoop/spark/jars/spark-1.3.1/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 364, in send_command
  File "/home/cheolsoop/spark/jars/spark-1.3.1/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 473, in send_command
  File "/usr/lib/python2.7/socket.py", line 430, in readline
    data = recv(1)
KeyboardInterrupt

Issue Links

• duplicates: SPARK-10001 Allow Ctrl-C in spark-shell to kill running job
• is related to: SPARK-8170 Ctrl-C in pyspark shell doesn't kill running job
• is related to: SPARK-10001 Allow Ctrl-C in spark-shell to kill running job

Sub-Tasks

• Ctrl-C in pyspark shell doesn't kill running job: Resolved