[SPARK-6953] Speedup tests of PySpark, reduce logging

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: PySpark
• Assignee: Davies Liu
• Reporter: Davies Liu
• Created: 16/Apr/15 00:01
• Updated: 22/Apr/15 00:50
• Resolved: 22/Apr/15 00:50

Description

Right now, it take about 30 minutes to complete the PySpark tests with Python 2.6, python 3.4 and PyPy. It's better to decrease it.

Also when run pyspark/tests.py, the logging is pretty scaring (lots of exceptions), it will be nice to mute the exception when it's expected.

Issue Links

• links to: [Github] Pull Request #5427 (davies)
• links to: [Github] Pull Request #5605 (rxin)