[IMPALA-273] Impalad runs into problem: "Lost connection to the state-store, entering recovery mode" often on all test clusters

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 0.7
• Fix Version/s: Impala 1.0
• Assignee: Henry Robinson
• Reporter: Lenni Kuff
• Created: 17/Apr/13 00:39
• Updated: 22/Apr/13 20:56
• Resolved: 22/Apr/13 20:56

Description

Impalad runs into problem: "Lost connection to the state-store, entering recovery mode" often on all test clusters. It happens more often in stress but we have also seen it happen when executing a single query. It was hit with both the old and new (v2) statestore implementation.

W0103 14:37:08.784693 5895 state-store-subscriber.cc:383] Lost connection to the state-store, entering recovery mode
I0103 14:37:09.703059 5895 state-store-subscriber.cc:85] Attempting to register service impala_backend_service on address 172.20.90.42:22000, to subscriber at 172.20.90.42:23000
I0103 14:37:09.766360 5895 state-store-subscriber.cc:137] Attempting to register subscriber for services impala_backend_service at 172.20.90.42:23000
I0103 14:37:09.904667 5895 state-store-subscriber.cc:137] Attempting to register subscriber for services impala_backend_service at 172.20.90.42:23000
W0103 14:37:20.016146 5895 state-store-subscriber.cc:383] Lost connection to the state-store, entering recovery mode
I0103 14:37:20.321495 5895 state-store-subscriber.cc:85] Attempting to register service impala_backend_service on address 172.20.90.42:22000, to subscriber at 172.20.90.42:23000
I0103 14:37:20.336611 5895 state-store-subscriber.cc:137] Attempting to register subscriber for services impala_backend_service at 172.20.90.42:23000
I0103 14:37:20.495632 5895 state-store-subscriber.cc:137] Attempting to register subscriber for services impala_backend_service at 172.20.90.42:23000
W0103 14:37:30.557159 5895 state-store-subscriber.cc:383] Lost connection to the state-store, entering recovery mode
I0103 14:37:31.324985 5895 state-store-subscriber.cc:85] Attempting to register service impala_backend_service on address 172.20.90.42:22000, to subscriber at 172.20.90.42:23000
I0103 14:37:33.255113 5895 state-store-subscriber.cc:137] Attempting to register subscriber for services impala_backend_service at 172.20.90.42:23000
I0103 14:37:33.586236 5895 state-store-subscriber.cc:137] Attempting to register subscriber for services impala_backend_service at 172.20.90.42:23000