[ACCUMULO-4663] ShutdownTServer attempts shutdown over and over again, can end up blocking migrations

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.7.2
• Assignee: Unassigned
• Reporter: John Vines
• Created: 21/Jun/17 17:29
• Updated: 26/Jan/21 03:15
• Resolved: 26/Jan/21 03:15

Description

Also affects 1.7.1

ACCUMULO-1259 identified a problem with it repeatedly invoking master.shutdownTServer. One side effect of this is a race where a server goes down, gets removed from the online tablet sets, etc. and then gets re-added to the serversToShutdown set. This will cause the balancer to not balance due to shutdown in progress and never gets rectified. Only workaround is to restart the master (or bring that server back up, I'm guessing).

ACCUMULO-3897 attempted to fix that problem by attempting shutdown once and only once. It does this by setting a local boolean. But because we do not reserialize our fate repos between isReady calls, this boolean effectively is reset between each check, making it pointless.

I believe there are 2 problems here- 1 is that ShutdownTServer.requestedShutdown is not implemented correctly
2 is we should have a mechanism to remove from serversToShutdown any server that is not present.