[IMPALA-12745] dump_breakpad_symbols.py's parallelism doesn't work with RPM/DEBs

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.4.0
• Fix Version/s: Impala 4.4.0
• Component/s: Infrastructure
• Epic Color: ghx-label-9
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 23/Jan/24 07:56
• Updated: 29/Jan/24 23:51
• Resolved: 29/Jan/24 23:46

Description

When using the "r" or "-pkg" option, dump_breakpad_symbols.py is extracting the RPM/DEB into a temporary directory. The lifetime of that temporary directory is maintained by yielding tuples from enumerate_pkg_files(). When using parallelism (added in IMPALA-11511), the yield doesn't keep the temporary directory around while the parallel threads are processing, so they fail with:

 

Found debugging info in /tmp/tmpqfZ9MZ/usr/lib/debug/usr/lib/impala/sbin-retail/impalad.debug
Failed to open ELF file '/tmp/tmpqfZ9MZ/usr/lib/debug/usr/lib/impala/sbin-retail/impalad.debug': No such file or directory
Failed to write symbol file.

 

Testing shows that this is still a problem with num_processes=1, so there should also be a change to be able to turn off the ThreadPool entirely. Processing OS packages can force the parallelism off for now as they don't benefit much from parallelism.

Issue Links

• is caused by: IMPALA-10048 Speed up dumping breadpad symbols in bin/jenkins/finalize.sh