[IMPALA-11401] Catalogd should log the table names causing OOM on array limit

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.2.0, Impala 4.1.1
• Component/s: Catalog
• Epic Color: ghx-label-6
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 28/Jun/22 13:17
• Updated: 30/Aug/22 12:10
• Resolved: 08/Aug/22 23:51

Description

Catalogd could throw OOM errors when serializing thrift objects larger than the 2GB byte array limit, e.g. in serializing responses of execDdl, getPartialCatalogObject requests. Such kind of OOM errors are table-level failures. They don't mean the server is running out of memory. Catalogd is still able to process other RPC requests.

To improve the supportability of catalogd, it should log the table name and some details of the request when throwing such OOM errors. Currently the log contains no details:

I0617 04:00:24.341722 534809 jni-util.cc:288] java.lang.OutOfMemoryError
        at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
        at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
        at org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:202)
        at org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.write(FieldSchema.java:531)
        at org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.write(FieldSchema.java:480)
        at org.apache.hadoop.hive.metastore.api.FieldSchema.write(FieldSchema.java:418)
        at org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1451)
        at org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.write(StorageDescriptor.java:1278)
        at org.apache.hadoop.hive.metastore.api.StorageDescriptor.write(StorageDescriptor.java:1144)
        at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:1414)
        at org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.write(Partition.java:1238)
        at org.apache.hadoop.hive.metastore.api.Partition.write(Partition.java:1099)
        at org.apache.impala.thrift.TPartialPartitionInfo$TPartialPartitionInfoStandardScheme.write(TPartialPartitionInfo.java:862)
        at org.apache.impala.thrift.TPartialPartitionInfo$TPartialPartitionInfoStandardScheme.write(TPartialPartitionInfo.java:759)
        at org.apache.impala.thrift.TPartialPartitionInfo.write(TPartialPartitionInfo.java:665)
        at org.apache.impala.thrift.TPartialTableInfo$TPartialTableInfoStandardScheme.write(TPartialTableInfo.java:914)
        at org.apache.impala.thrift.TPartialTableInfo$TPartialTableInfoStandardScheme.write(TPartialTableInfo.java:790)
        at org.apache.impala.thrift.TPartialTableInfo.write(TPartialTableInfo.java:688)
        at org.apache.impala.thrift.TGetPartialCatalogObjectResponse$TGetPartialCatalogObjectResponseStandardScheme.write(TGetPartialCatalogObjectResponse.java:977)
        at org.apache.impala.thrift.TGetPartialCatalogObjectResponse$TGetPartialCatalogObjectResponseStandardScheme.write(TGetPartialCatalogObjectResponse.java:857)
        at org.apache.impala.thrift.TGetPartialCatalogObjectResponse.write(TGetPartialCatalogObjectResponse.java:739)
        at org.apache.thrift.TSerializer.serialize(TSerializer.java:79)
        at org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:254)
I0617 04:00:24.341833 534809 status.cc:129] OutOfMemoryError: null
    @           0xc1d7f3
    @          0x13a8679
    @           0xc05bf3
    @           0xbf3433
    @           0xd133c4
    @           0xd07d33
    @           0xd1f3a2
    @          0x10d214a
    @          0x10c5702
    @          0x144de71
    @          0x144f2ea
    @          0x1c9f8d1
    @     0x7ff6aca68ea4
    @     0x7ff6a9540b0c
E0617 04:00:24.341909 534809 catalog-server.cc:209] OutOfMemoryError: null

Issue Links

• links to: Code Review