[SPARK-32363] Flaky pip installation test in Jenkins

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.1.0
• Fix Version/s: 3.0.1, 3.1.0
• Component/s: PySpark
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 20/Jul/20 04:48
• Updated: 12/Dec/22 18:10
• Resolved: 21/Jul/20 13:50

Description

Currently pip packaging test is flaky in Jenkins:

Installing collected packages: py4j, pyspark
  Attempting uninstall: py4j
    Found existing installation: py4j 0.10.9
    Uninstalling py4j-0.10.9:
      Successfully uninstalled py4j-0.10.9
  Attempting uninstall: pyspark
    Found existing installation: pyspark 3.1.0.dev0
ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/tmp.GX6lHKLHZK/3.6/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/tmp/tmp.GX6lHKLHZK/3.6/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "/tmp/tmp.GX6lHKLHZK/3.6/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 407, in run
    use_user_site=options.use_user_site,
  File "/tmp/tmp.GX6lHKLHZK/3.6/lib/python3.6/site-packages/pip/_internal/req/__init__.py", line 64, in install_given_reqs
    auto_confirm=True
  File "/tmp/tmp.GX6lHKLHZK/3.6/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 675, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "/tmp/tmp.GX6lHKLHZK/3.6/lib/python3.6/site-packages/pip/_internal/req/req_uninstall.py", line 545, in from_dist
    link_pointer, dist.project_name, dist.location)
AssertionError: Egg-link /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-2.3/python does not match installed location of pyspark (at /home/jenkins/workspace/SparkPullRequestBuilder@2/python)
Cleaning up temporary directory - /tmp/tmp.GX6lHKLHZK

Issue Links

• relates to: SPARK-32316 Test PySpark with Python 3.8 in Github Actions as well
• links to: [Github] Pull Request #29117 (HyukjinKwon)
• links to: [Github] Pull Request #29117 (HyukjinKwon)
• links to: [Github] Pull Request #29215 (HyukjinKwon)