[IMPALA-12080] Test test_recover_many_partitions is very slow on S3, Ozone

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Component/s: Infrastructure
• Epic Color: ghx-label-8
• Assignee: Michael Smith
• Reporter: Michael Smith
• Created: 21/Apr/23 21:16
• Updated: 24/Apr/23 17:33
• Resolved: 24/Apr/23 17:33

Description

The test metadata/test_recover_partitions.py::TestRecoverPartitions::test_recover_many_partitions takes <2 minutes for HDFS runs, ~1 hour for Ozone and S3 tests. This appears to be because test setup invokes a filesystem client 1400 times, to create directories and files for 700 partitions. With HDFS that's fast because it uses pywebhdfs to do it in-process, but for all other filesystems it defaults to the hdfs Java CLI.