[IMPALA-2045] catalogd remains up. Stops after connecting to metastore. Never proceeds to listen on port 26000

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Workaround
• Affects Version/s: Impala 2.1.1
• Component/s: Catalog
• Environment: Centos release 6.3; CDH 5.3.3 installed with packages
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Rohit Nijhawan
• Created: 01/Jun/15 22:24
• Updated: 19/Jan/16 22:19
• Resolved: 19/Jan/16 22:19

Description

Hello,

I've struggled with this for a few weeks. The symptom is that catalogd starts, logs its output but doesn't proceed past

I0601 17:23:25.342360 10846 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 4
I0601 17:23:25.342622 10846 HiveMetaStoreClient.java:308] Trying to connect to metastore with URI thrift://localhost.localdomain:10000
I0601 17:23:25.342890 10846 HiveMetaStoreClient.java:396] Connected to metastore.

• I launch the state store first and it seems to be running and listening on all ports.
• Log Level 2 using GLOG_v is set through /etc/default/impala
• Can't create database. Impala says statestore isn't updated.
• Invalidate metadata in impala-shell fails with

  [localhost.localdomain:21000] > invalidate metadata;
  Query: invalidate metadata
  ERROR: Couldn't open transport for 127.0.0.1:26000 (connect() failed: Connection refused)

• Impala version

  [training@localhost ~]$ sudo yum list installed | grep impala
  impala.x86_64                    2.1.3+cdh5.3.3+0-1.cdh5.3.3.p0.8.el6
  impala-catalog.x86_64            2.1.3+cdh5.3.3+0-1.cdh5.3.3.p0.8.el6
  impala-server.x86_64             2.1.3+cdh5.3.3+0-1.cdh5.3.3.p0.8.el6
  impala-shell.x86_64              2.1.3+cdh5.3.3+0-1.cdh5.3.3.p0.8.el6
  impala-state-store.x86_64        2.1.3+cdh5.3.3+0-1.cdh5.3.3.p0.8.el6
  

• So my question is: what can cause catalogd to stop execution but remain up and not proceed to become a service on port 26000 ?

Log files are attached. Any other assistance from you would be greatly appreciated.
-Rohit