[SPARK-23292] python tests related to pandas are skipped with python 2

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Tests
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Yin Huai
• Created: 01/Feb/18 04:26
• Updated: 08/Oct/19 05:41
• Resolved: 08/Oct/19 05:41

Description

I was running python tests and found that pyspark.sql.tests.GroupbyAggPandasUDFTests.test_unsupported_types does not run with Python 2 because the test uses "assertRaisesRegex" (supported by Python 3) instead of "assertRaisesRegexp" (supported by Python 2). However, spark jenkins does not fail because of this issue (see run history at here). After looking into this issue, seems test script will skip tests related to pandas if pandas is not installed, which means that jenkins does not have pandas installed. 
 
Since pyarrow related tests have the same skipping logic, we will need to check if jenkins has pyarrow installed correctly as well. 
 
Since features using pandas and pyarrow are in 2.3, we should fix the test issue and make sure all tests pass before we make the release.

Issue Links

• is related to: SPARK-23300 Print out if Pandas and PyArrow are installed or not in tests
• links to: [Github] Pull Request #20465 (cloud-fan)