[IMPALA-6683] Restarting the Catalog without restarting Impalad and SS can block topic updates

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.12.0
• Fix Version/s: Impala 3.0, Impala 2.12.0
• Component/s: Catalog
• Epic Color: ghx-label-7
• Assignee: Tianyi Wang
• Reporter: Mostafa Mokhtar
• Created: 16/Mar/18 00:25
• Updated: 20/Mar/18 20:44
• Resolved: 16/Mar/18 20:29

Description

After restarting the Catalog without restarting the other services topic updates are getting stuck

From SS

I0315 17:20:44.305951 120990 statestore.cc:242] Preparing initial catalog-update topic update for [e-mail]:22000. Size = 50.18 MB
I0315 17:20:44.305981 120974 statestore.cc:242] Preparing initial catalog-update topic update for [e-mail]:22000. Size = 50.18 MB
I0315 17:20:44.305982 120975 statestore.cc:242] Preparing initial catalog-update topic update for [e-mail]:22000. Size = 50.18 MB
I0315 17:20:44.343500 121001 statestore.cc:652] Received request for different delta base of topic: catalog-update from: [e-mail]:22000 subscriber from_version: 0
I0315 17:20:44.355831 120963 statestore.cc:242] Preparing initial catalog-update topic update for [e-mail]:22000. Size = 50.18 MB
I0315 17:20:44.355840 120983 statestore.cc:242] Preparing initial catalog-update topic update for [e-mail]:22000. Size = 50.18 MB

From Impalad

E0315 17:20:36.468525  7355 impala-server.cc:1381] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
E0315 17:20:40.778920  7355 impala-server.cc:1381] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
I0315 17:20:42.951370 11846 StmtMetadataLoader.java:193] Waiting for table metadata. Waited for 170 catalog updates and 340149ms. Tables remaining: [metadata_benchmarks.80_partitions_250k_files]
I0315 17:21:04.954602 11846 StmtMetadataLoader.java:214] Re-sending prioritized load request. Waited for 180 catalog updates and 362152ms.
I0315 17:21:04.955175 11846 FeSupport.java:274] Requesting prioritized load of table(s): metadata_benchmarks.80_partitions_250k_files

Issue Links

• is broken by: IMPALA-5990 End-to-end compression of metadata