[ARROW-10457] [CI] Fix Spark branch-3.0 integration tests

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: Continuous Integration, Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26434
• Assignee: Bryan Cutler
• Reporter: Bryan Cutler
• Created: 02/Nov/20 05:21
• Updated: 11/Jan/23 08:13
• Resolved: 05/Feb/21 00:27

Description

The Spark branch-3.0 is currently failing because this branch has not been updated or patched to use the latest Arrow Java, see https://github.com/ursa-labs/crossbow/actions?query=branch:actions-681-github-test-conda-python-3.7-spark-branch-3.0. The Spark branch-3.0 has already been released and only able to get bug fixes. Instead of patching the Spark build, we should not try to rebuild Spark with the latest Arrow Java, and instead only test against the latest pyarrow. This should work, but might also need a minor Python patch.

Issue Links

• links to: GitHub Pull Request #9115
• links to: GitHub Pull Request #9210