[STORM-2651] Executor stopped working after connectivity issues with ZK. Executor is not restarted by nimbus scheduler.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.2
• Component/s: storm-core
• Assignee: Unassigned
• Reporter: Ashok
• Created: 21/Jul/17 09:38
• Updated: 21/Jul/17 09:38

Description

After connectivity issues, nimbus scheduler assigned the appropriate executor to the slots(custom scheduler).

{panel:title=My title}
o.a.s.d.nimbus [INFO] Setting new assignment for topology id <<topology_name>>-1499356635: #org.apache.storm.daemon.common.Assignment{:master-code-dir "/opt/storm_datadir", :node->host {"c97a7a58-ec31-41a6-8585-43ef7b62ea83" "test1", "4ec038e7-281b-4dcc-9e70-57afa1fd84c4" "test2", "c13b0fc8-d5c1-4335-8339-17b3c048b160" "test3", "0b8b056a-dda1-4d32-8c74-003a1fefad7e" "test4"}, :executor->node+port {[8 8] ["0b8b056a-dda1-4d32-8c74-003a1fefad7e" 6703], [12 12] ["c13b0fc8-d5c1-4335-8339-17b3c048b160" 6702], [2 2] ["0b8b056a-dda1-4d32-8c74-003a1fefad7e" 6703], [7 7] ["4ec038e7-281b-4dcc-9e70-57afa1fd84c4" 6702], [22 22] ["c13b0fc8-d5c1-4335-8339-17b3c048b160" 6702], [3 3] ["c13b0fc8-d5c1-4335-8339-17b3c048b160" 6702], [24 24] ["4ec038e7-281b-4dcc-9e70-57afa1fd84c4" 6702], [1 1] ["4ec038e7-281b-4dcc-9e70-57afa1fd84c4" 6702], [18 18] ["c97a7a58-ec31-41a6-8585-43ef7b62ea83" 6703], [6 6] ["c97a7a58-ec31-41a6-8585-43ef7b62ea83" 6703], [20 20] ["4ec038e7-281b-4dcc-9e70-57afa1fd84c4" 6702], [9 9] ["0b8b056a-dda1-4d32-8c74-003a1fefad7e" 6703], [23 23] ["c97a7a58-ec31-41a6-8585-43ef7b62ea83" 6703], [11 11] ["c97a7a58-ec31-41a6-8585-43ef7b62ea83" 6703], [16 16] ["4ec038e7-281b-4dcc-9e70-57afa1fd84c4" 6702], [13 13] ["c97a7a58-ec31-41a6-8585-43ef7b62ea83" 6703], [19 19] ["0b8b056a-dda1-4d32-8c74-003a1fefad7e" 6703], [21 21] ["0b8b056a-dda1-4d32-8c74-003a1fefad7e" 6703], [5 5] ["c13b0fc8-d5c1-4335-8339-17b3c048b160" 6702], [10 10] ["4ec038e7-281b-4dcc-9e70-57afa1fd84c4" 6702], [14 14] ["c13b0fc8-d5c1-4335-8339-17b3c048b160" 6702], [4 4] ["c97a7a58-ec31-41a6-8585-43ef7b62ea83" 6703], [15 15] ["0b8b056a-dda1-4d32-8c74-003a1fefad7e" 6703], [17 17] ["c13b0fc8-d5c1-4335-8339-17b3c048b160" 6702]}, :executor->start-time-secs {[8 8] 1499356646, [12 12] 1499356646, [2 2] 1499356646, [7 7] 1499356646, [22 22] 1499356646, [3 3] 1499356646, [24 24] 1499356646, [1 1] 1499356646, [18 18] 1499356646, [6 6] 1499356646, [20 20] 1499356646, [9 9] 1499356646, [23 23] 1499356646, [11 11] 1499356646, [16 16] 1499356646, [13 13] 1499356646, [19 19] 1499356646, [21 21] 1499356646, [5 5] 1499356646, [10 10] 1499356646, [14 14] 1499356646, [4 4] 1499356646, [15 15] 1499356646, [17 17] 1499356646}, :worker->resources {["c13b0fc8-d5c1-4335-8339-17b3c048b160" 6702] [0.0 0.0 0.0], ["4ec038e7-281b-4dcc-9e70-57afa1fd84c4" 6702] [0.0 0.0 0.0], ["c97a7a58-ec31-41a6-8585-43ef7b62ea83" 6703] [0.0 0.0 0.0], ["0b8b056a-dda1-4d32-8c74-003a1fefad7e" 6703] [0.0 0.0 0.0]}}
{panel}

Then all the executor are started working properly.

When I checked in-depth I found that, one of the spout executor has not started and also found that nimbus stopped logging after this issue.

o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with key<<topology-name >>-1499356635-stormconf.ser