[RATIS-622] createLog will block indefinitely if group creation fails

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: LogService
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 12/Jul/19 20:46
• Updated: 16/Apr/20 02:03
• Resolved: 16/Apr/20 02:03

Description

I've seen this bug a few times, but finally got to the bottom of it. The VerificationTool will get to the end, but not exit. The reason is that at least one log has still not yet been created.

• The client will be blocked asking the metadataquorum to create a log
• The metadata leader will have tons of threads blocked on the writeLock

What I just found today was an exception, logged to stderr (not stdout), that was thrown when the group create failed.

However, we don't catch and re-throw this exception on group create failure, but proceed then to try to send a message to the group which failed to be created.

We must catch this failure, try to clean up, and propagate the original createLog exception back to the client.