[FLUME-647] After the master has been running for a while (like 3 weeks) adding 6 new flume-nodes causes it to hang, then lose config on restart

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.9.3
• Fix Version/s: 0.9.5
• Component/s: Master
• Labels: configuration flume

Environment

redhat linux using the official RPM

• Assignee: Unassigned
• Reporter: Disabled imported user
• Created: 27/May/11 23:40
• Updated: 05/Nov/14 11:18
• Resolved: 05/Nov/14 11:18

Description

Here's what happened to me:
0) Check the node statuses on the flume master, realize I'm missing 6 servers
1) Start up 6 brand new flume-nodes, point them at the flume master
2) log on to the master to configure them
– at this point I can't get a response from the master! I can connect via the shell, but every query I execute hangs. Browsing to the web dashboard hangs too.

3) "oh crap, no problem I can restart it"
4) restart the flume master
5) flume master comes back online, but it has lost EVERY configuration
6) I scramble to reconfigure every node using a ruby script

I realize I should probably have more than one master process, but all I did was add new nodes to the cluster!