[IMPALA-2648] catalogd crashes when serialized messages are over 2 GB

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.3.0
• Fix Version/s: Impala 2.12.0
• Component/s: Catalog
• Labels: compute-stats crash downgraded
• Target Version: Product Backlog
• Assignee: Tianyi Wang
• Reporter: Silvius Rus
• Created: 08/Nov/15 18:05
• Updated: 03/Dec/18 20:44
• Resolved: 03/Dec/18 20:44

Description

We've seen a catalogd crash triggered by loading the metadata for a table with about 20K partitions and 77 columns that has incremental stats. It looks like the serialized message is over 2GB which is the Java max array size. Ideally we should catch this exception and fail the query that needs this table's metadata with an appropriate message.

I1107 06:47:56.641507 30252 jni-util.cc:177] java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
at org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:187)
at com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1831)
at com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1543)
at com.cloudera.impala.thrift.THdfsPartition.write(THdfsPartition.java:1389)
at com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:1123)
at com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:969)
at com.cloudera.impala.thrift.THdfsTable.write(THdfsTable.java:848)
at com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1628)
at com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1395)
at com.cloudera.impala.thrift.TTable.write(TTable.java:1209)
at com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1241)
at com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1098)
at com.cloudera.impala.thrift.TCatalogObject.write(TCatalogObject.java:938)
at com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:487)
at com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:421)
at com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse.write(TGetAllCatalogObjectsResponse.java:365)
at org.apache.thrift.TSerializer.serialize(TSerializer.java:79)
at com.cloudera.impala.service.JniCatalog.getCatalogObjects(JniCatalog.java:110)

You can identify this issue by looking at the metastore database. Here is how to see the size of the incremental stats for table id 12345. The table with this value of 624 MB of incremental stats led to the catalogd crash shown above.

postgres=# select pg_size_pretty(sum(length("PARTITION_PARAMS"."PARAM_KEY") + length("PARTITION_PARAMS"."PARAM_VALUE"))) from "PARTITIONS", "PARTITION_PARAMS" where "PARTITIONS"."TBL_ID"=12345 and "PARTITIONS"."PART_ID" = "PARTITION_PARAMS"."PART_ID"  and "PARTITION_PARAMS"."PARAM_KEY" LIKE 'impala_intermediate%';
 pg_size_pretty 
----------------
 624 MB
(1 row)

Issue Links

• Blocked: IMPALA-5990 End-to-end compression of metadata
• is blocked by: IMPALA-2649 improve incremental stats scalability
• is related to: IMPALA-3910 catalogd becomes unusable when Java array size limit hit.
• relates to: IMPALA-7424 Improve in-memory representation of incremental stats
• relates to: IMPALA-3961 java.io.ByteArrayOutputStream unable to allocate byte array with Integer.MAX_VALUE size