[ACCUMULO-2645] tablet stuck unloading, and problem is hard to diagnose

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.4
• Fix Version/s: 1.6.1, 1.7.0
• Component/s: tserver
• Labels: newbie

Environment

very large production cluster, CDH3u5

• Assignee: Eric C. Newton
• Reporter: Eric C. Newton
• Created: 07/Apr/14 19:45
• Updated: 12/Sep/14 22:37
• Resolved: 12/Sep/14 17:52

Description

• master failed to balance
• custom balancer refused to balance while migrations were in place
• tablet server was not unloading the tablet
• tablet server was otherwise serving tablets, providing status
• memory dump determined that there were 21K UnloadTabletHandler objects
• jstack showed UnloadTabletHandler in Tablet.completeClose, line 2674
• the last print of the debug "completeClose(safeState=true, completeClose=true) occured 9 days ago
• there was a query that had been running for 9 days

Issue Links

• relates to: ACCUMULO-2673 Random walk balance check is still failing too frequently