[IMPALA-6683] Restarting the Catalog without restarting Impalad and SS can block topic updates - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Blocker
Resolution: Fixed
Affects Version/s: Impala 2.12.0
Fix Version/s: Impala 3.0, Impala 2.12.0
Component/s: Catalog
Labels:
None

Epic Color:
ghx-label-7

Description

After restarting the Catalog without restarting the other services topic updates are getting stuck

From SS

I0315 17:20:44.305951 120990 statestore.cc:242] Preparing initial catalog-update topic update for impalad@va1031.foo:22000. Size = 50.18 MB
I0315 17:20:44.305981 120974 statestore.cc:242] Preparing initial catalog-update topic update for impalad@vd1120.foo:22000. Size = 50.18 MB
I0315 17:20:44.305982 120975 statestore.cc:242] Preparing initial catalog-update topic update for impalad@vc1518.foo:22000. Size = 50.18 MB
I0315 17:20:44.343500 121001 statestore.cc:652] Received request for different delta base of topic: catalog-update from: impalad@vd1107.foo:22000 subscriber from_version: 0
I0315 17:20:44.355831 120963 statestore.cc:242] Preparing initial catalog-update topic update for impalad@ve1134.foo:22000. Size = 50.18 MB
I0315 17:20:44.355840 120983 statestore.cc:242] Preparing initial catalog-update topic update for impalad@vd1329.foo:22000. Size = 50.18 MB

From Impalad

E0315 17:20:36.468525  7355 impala-server.cc:1381] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
E0315 17:20:40.778920  7355 impala-server.cc:1381] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
I0315 17:20:42.951370 11846 StmtMetadataLoader.java:193] Waiting for table metadata. Waited for 170 catalog updates and 340149ms. Tables remaining: [metadata_benchmarks.80_partitions_250k_files]
I0315 17:21:04.954602 11846 StmtMetadataLoader.java:214] Re-sending prioritized load request. Waited for 180 catalog updates and 362152ms.
I0315 17:21:04.955175 11846 FeSupport.java:274] Requesting prioritized load of table(s): metadata_benchmarks.80_partitions_250k_files