[HBASE-24493] [flakey test] TestExportSnapshot family of tests failing due to timeout in AbstractDelegationTokenSecretManager$ExpiredTokenRemover

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 3.0.0-alpha-1, 2.3.0
• Component/s: test
• Assignee: Michael Stack
• Reporter: Nick Dimiduk
• Created: 02/Jun/20 17:17
• Updated: 22/Jun/20 23:53
• Resolved: 08/Jun/20 21:21

Description

I've observed another occurrence of this test timing out, over on https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/3/

The failure message is cryptic, but I think i found the issue. This isn't the HBaseClassTestRule invoking the timeout, it's something in the test, I think in the mini-cluster.

There appears to be a timeout set in some kind of secret manager, which is too aggressive for this mini-cluster test. The last component of the mini-cluster, MapReduce, is finally available at T+273501ms – ~4.5 minutes after process launch. This is how I interpret the log line

2020-06-02 03:20:49,252 INFO  [Thread-223] server.Server(419): Started @273501ms

a scant 20ms later we get

2020-06-02 03:20:50,274 ERROR [Thread[Thread-224,5,FailOnTimeoutGroup]] delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover(700): ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
2020-06-02 03:20:50,351 INFO  [Time-limited test] hbase.HBaseTestingUtility(1272): Shutting down minicluster

These thread group names have no meaning to me.

Issue Links

• relates to: HBASE-24131 [Flakey Tests] TestExportSnapshot takes too long; up against 13min max
• links to: GitHub Pull Request #1832

Sub-Tasks

• Reenable the TestExportSnapshot family of tests: Open