[HBASE-24001] The "HBase Nightly" job may exhaust all executors - and create a deadlock

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: jenkins
• Assignee: Michael Stack
• Reporter: Zoltan Haindrich
• Created: 17/Mar/20 11:03
• Updated: 30/Sep/24 02:15
• Resolved: 30/Mar/20 08:45

Description

What seems to happen is:

• the job needs more and more executors to run
• in case there are several of them launched in parallel (different branches) say 5
• they take up all available executors labeled "Hadoop" and then they wait for more indefinetly

I've killed a few builds...see my notes here:

https://the-asf.slack.com/archives/C7M1AFYLS/p1584442366000900

Issue Links

• is related to: HBASE-24017 Turn down flakey rerun rate on all but hot branches
• relates to: HBASE-24016 Change nightly poll from cron @daily to pollSCM @daily; i.e. run nightly if a change ONLY