[IMPALA-10076] Reduce logs about partition level catalog updates

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0
• Epic Link: fine-grained metadata in catalogd
• Epic Color: ghx-label-8
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 12/Aug/20 09:20
• Updated: 19/Sep/20 07:30
• Resolved: 19/Sep/20 07:30

Description

After IMPALA-3127, catalogd will send catalog updates in partition level. So catalogd will log updates in partition level, e.g.

I0812 17:12:18.367053 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=6, version=1451, original size=702, compressed size=529
I0812 17:12:18.367229 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=7, version=1451, original size=702, compressed size=530
I0812 17:12:18.367377 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=8, version=1451, original size=702, compressed size=528
I0812 17:12:18.367514 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=9, version=1451, original size=702, compressed size=530
I0812 17:12:18.367652 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=1, version=1451, original size=702, compressed size=530
I0812 17:12:18.367787 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=10, version=1451, original size=704, compressed size=527
I0812 17:12:18.367918 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=11, version=1451, original size=704, compressed size=530
I0812 17:12:18.368062 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=12, version=1451, original size=704, compressed size=532
I0812 17:12:18.368203 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=2, version=1451, original size=702, compressed size=529
I0812 17:12:18.368335 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=3, version=1451, original size=702, compressed size=528
I0812 17:12:18.368495 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=4, version=1451, original size=702, compressed size=527
I0812 17:12:18.368629 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=5, version=1451, original size=702, compressed size=525
I0812 17:12:18.368762 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=6, version=1451, original size=702, compressed size=529
I0812 17:12:18.368984 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=7, version=1451, original size=702, compressed size=530
I0812 17:12:18.369117 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=8, version=1451, original size=702, compressed size=529
I0812 17:12:18.369246 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2010/month=9, version=1451, original size=702, compressed size=530
I0812 17:12:18.369390 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=1, version=1451, original size=702, compressed size=529
I0812 17:12:18.369525 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=10, version=1451, original size=704, compressed size=525
I0812 17:12:18.369654 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=11, version=1451, original size=704, compressed size=530
I0812 17:12:18.369783 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=12, version=1451, original size=704, compressed size=533
I0812 17:12:18.369912 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=2, version=1451, original size=702, compressed size=528
I0812 17:12:18.370040 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=3, version=1451, original size=702, compressed size=528
I0812 17:12:18.370168 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=4, version=1451, original size=702, compressed size=527
I0812 17:12:18.370322 29433 catalog-server.cc:737] Collected update: 1:HDFS_PARTITION:functional.alltypes:year=2009/month=5, version=1451, original size=702, compressed size=526
I0812 17:12:18.371448 29433 catalog-server.cc:737] Collected update: 1:TABLE:functional.alltypes, version=1451, original size=3944, compressed size=1859
I0812 17:12:18.373958 29433 catalog-server.cc:737] Collected update: 1:CATALOG_SERVICE_ID, version=1451, original size=60, compressed size=58
I0812 17:12:20.364428 29443 catalog-server.cc:342] A catalog update with 26 entries is assembled. Catalog version: 1451 Last sent catalog version: 1450

Similarly, coordinators will log them as well, e.g.

I0812 17:12:21.240887 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=6 version: 1451 size: 702
I0812 17:12:21.241179 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=7 version: 1451 size: 702
I0812 17:12:21.241423 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=8 version: 1451 size: 702
I0812 17:12:21.241655 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=9 version: 1451 size: 702
I0812 17:12:21.241880 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=1 version: 1451 size: 702
I0812 17:12:21.242110 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=10 version: 1451 size: 704
I0812 17:12:21.242337 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=11 version: 1451 size: 704
I0812 17:12:21.242563 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=12 version: 1451 size: 704
I0812 17:12:21.242816 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=2 version: 1451 size: 702
I0812 17:12:21.243049 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=3 version: 1451 size: 702
I0812 17:12:21.243273 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=4 version: 1451 size: 702
I0812 17:12:21.243499 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=5 version: 1451 size: 702
I0812 17:12:21.243721 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=6 version: 1451 size: 702
I0812 17:12:21.243942 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=7 version: 1451 size: 702
I0812 17:12:21.244164 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=8 version: 1451 size: 702
I0812 17:12:21.244388 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2010/month=9 version: 1451 size: 702
I0812 17:12:21.244612 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=1 version: 1451 size: 702
I0812 17:12:21.244838 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=10 version: 1451 size: 704
I0812 17:12:21.245064 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=11 version: 1451 size: 704
I0812 17:12:21.245330 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=12 version: 1451 size: 704
I0812 17:12:21.245561 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=2 version: 1451 size: 702
I0812 17:12:21.245918 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=3 version: 1451 size: 702
I0812 17:12:21.246156 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=4 version: 1451 size: 702
I0812 17:12:21.246387 30308 ImpaladCatalog.java:222] Adding: HDFS_PARTITION:functional.alltypes:year=2009/month=5 version: 1451 size: 702
I0812 17:12:21.247385 30308 ImpaladCatalog.java:222] Adding: TABLE:functional.alltypes version: 1451 size: 3944 
I0812 17:12:21.247516 30308 ImpaladCatalog.java:222] Adding: CATALOG_SERVICE_ID version: 1451 size: 60
I0812 17:12:21.248576 30308 impala-server.cc:1770] Catalog topic update applied with version: 1451 new min catalog object version: 3

These are too verbose. Especially for tables with many partitions. We should find a way to reduce these logs.

Issue Links

• links to: Code Review