[HBASE-24001] The "HBase Nightly" job may exhaust all executors - and create a deadlock

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.0
• Assignee: Michael Stack
• Reporter: Zoltan Haindrich
• Created: 17/Mar/20 11:03
• Updated: 07/Apr/20 23:25
• Resolved: 30/Mar/20 08:45

Description

What seems to happen is:

• the job needs more and more executors to run
• in case there are several of them launched in parallel (different branches) say 5
• they take up all available executors labeled "Hadoop" and then they wait for more indefinetly

I've killed a few builds...see my notes here:

https://the-asf.slack.com/archives/C7M1AFYLS/p1584442366000900

Issue Links

• is related to: HBASE-24017 Turn down flakey rerun rate on all but hot branches
• relates to: HBASE-24016 Change nightly poll from cron @daily to pollSCM @daily; i.e. run nightly if a change ONLY