[IMPALA-10048] Speed up dumping breadpad symbols in bin/jenkins/finalize.sh

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0
• Fix Version/s: Impala 4.4.0
• Component/s: Infrastructure
• Labels: ramp-up
• Epic Color: ghx-label-1
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 05/Aug/20 05:26
• Updated: 29/Jan/24 23:51
• Resolved: 08/Jan/24 19:17

Description

bin/jenkins/finalize.sh has code to find and symbolize breakpad minidumps. This process is very slow, because it runs bin/dump_breakpad_symbols.py for ${IMPALA_HOME}/be/build/latest, which includes all of the backend tests. There are two problems:

1. It is dumping symbols for a large number of binaries that are not useful unless the minidump is for that particular binary. (i.e. all the backend tests are useless if it is resolving an impalad minidump)
2. bin/dump_breakpad_symbols.py runs serially and could be much faster if it went parallel

We should fix one or both of these problems.

Issue Links

• causes: IMPALA-12745 dump_breakpad_symbols.py's parallelism doesn't work with RPM/DEBs