[IMPALA-6629] Clearer and more concise logging during catalog topic updates

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.12.0
• Fix Version/s: Impala 3.0, Impala 2.12.0
• Component/s: Catalog
• Labels: supportability
• Target Version: Impala 2.12.0
• Epic Color: ghx-label-3
• Assignee: Tianyi Wang
• Reporter: Alexander Behm
• Created: 08/Mar/18 05:39
• Updated: 12/Mar/18 22:07
• Resolved: 12/Mar/18 22:05

Description

We should improve the logging on the catalogd side during a catalog topic update.

Snippet of current log:

I0307 21:23:41.011883 21342 catalog-server.cc:477] Publishing update: TABLE:functional_avro_def.nulltable original size: 65
I0307 21:23:41.011922 21342 CatalogServiceCatalog.java:426] Collected catalog update: TABLE:functional_avro_def.alltypestiny version: 134
I0307 21:23:41.011927 21342 catalog-server.cc:477] Publishing update: TABLE:functional_avro_def.alltypestiny original size: 68
I0307 21:23:41.011975 21342 CatalogServiceCatalog.java:426] Collected catalog update: TABLE:functional_avro_def.jointbl version: 138
I0307 21:23:41.011986 21342 catalog-server.cc:477] Publishing update: TABLE:functional_avro_def.jointbl original size: 63
I0307 21:23:41.012027 21342 CatalogServiceCatalog.java:426] Collected catalog update: TABLE:functional_avro_def.alltypesaggmultifilesnopart version: 130
I0307 21:23:41.012032 21342 catalog-server.cc:477] Publishing update: TABLE:functional_avro_def.alltypesaggmultifilesnopart original size: 83
I0307 21:23:41.012071 21342 CatalogServiceCatalog.java:426] Collected catalog update: TABLE:functional_avro_def.alltypesaggnonulls version: 131

We should improve the logging as follows

• Avoid duplicate messages for the same entity
• Be clearer about "publishing" and "collecting" a topic update (collect is more accurate)
• Add a log final message when a topic update has been fully assembled

The double-logging was introduced in IMPALA-5990. For heavily loaded clusters this change might lead to a doubling of the log volume - clearly undesirable.