Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-965

Sudden problems with impalad and statestore

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Bug
    • Affects Version/s: Impala 1.2.4
    • Fix Version/s: None
    • Component/s: None
    • Environment:
      CDH 4.6 from Parcels, Impala 1.2.4 from Parcels
      CM 4.8.1

      Description

      Hi, we started to get strange bug with statestore and impalad
      Here is what we do.
      1. connect to local impalad
      2. show tables - OK
      3. select from table - FAIL, ERROR: No backends configured

      devops@prod-node012 ~]$ impala-shell 
      Starting Impala Shell without Kerberos authentication
      Connected to prod-node012.kyc.lol.ru:21000
      Server version: impalad version 1.2.4 RELEASE (build ac29ae09d66c1244fe2ceb293083723226e66c1a)
      Welcome to the Impala shell. Press TAB twice to see a list of available commands.
      
      Copyright (c) 2012 Cloudera, Inc. All rights reserved.
      
      (Shell build version: Impala Shell v1.2.4 (ac29ae0) built on Wed Mar  5 07:05:40 PST 2014)
      [prod-node012.kyc.lol.ru:21000] > connect localhost;
      Connected to localhost:21000
      Server version: impalad version 1.2.4 RELEASE (build ac29ae09d66c1244fe2ceb293083723226e66c1a)
      [localhost:21000] > show tables;
      Query: show tables
      +------------------------------------------------------------------+
      | name                                                             |
      +------------------------------------------------------------------+
      | my_fav_table                                                     |
      | many_more_goes_here                                              |
      +------------------------------------------------------------------+
      Returned 121 row(s) in 0.02s
      [localhost:21000] > select count(*) from my_fav_table;
      Query: select count(*) from my_fav_table
      ERROR: No backends configured
      

      Here is node012 impalad log:

      15:40:49.197	INFO	statestore-subscriber.cc:216	
      prod-node0102.kyc.lol.ru:22000: Connection with statestore lost, entering recovery mode
      15:40:49.197	INFO	statestore-subscriber.cc:220	
      Trying to re-register with statestore, attempt: 1
      15:42:25.235	INFO	statestore-subscriber.cc:159	
      Subscriber registration ID: 7f470575215239dc:63643be6ab1fb98c
      15:42:25.235	INFO	statestore-subscriber.cc:224	
      Reconnected to statestore. Exiting recovery mode
      15:42:55.236	INFO	statestore-subscriber.cc:216	
      prod-node0102.kyc.lol.ru:22000: Connection with statestore lost, entering recovery mode
      15:42:55.236	INFO	statestore-subscriber.cc:220	
      Trying to re-register with statestore, attempt: 1
      

      Here is statestore log:

      15:40:19.110	INFO	statestore.cc:329	
      Subscriber 'prod-node012.kyc.lol.ru:22000' registered (registration id: 98437526ee48f64f:7fcb9fc4796414af)
      
      15:40:49.111	INFO	statestore.cc:298	
      Registering: prod-node012.kyc.lol.ru:22000
      
      15:42:25.148	INFO	statestore.cc:329	
      Subscriber 'prod-node012.kyc.lol.ru:22000' registered (registration id: f41c966d3733c1c:655a2ccea68edebb)
      

      What it could be?
      We have more roles (TT, DN) running on prod-node012 and there are no problems with them. We did upgrade to Impala 1.2.4, CDH 4.6 a month ago and problem appeared today. I have no idea what to do. how can we debug it?

        Attachments

          Activity

            People

            • Assignee:
              henryr Henry Robinson
              Reporter:
              serega_sheypak Sergey
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: