[IMPALA-4902] Concurrent DDL may fail with a ConcurrentModificationException

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Impala 2.9.0
• Component/s: Catalog
• Labels: stress
• Target Version: Impala 2.9.0
• Assignee: Alexander Behm
• Reporter: Michael Brown
• Created: 08/Feb/17 18:51
• Updated: 08/Dec/17 23:08
• Resolved: 24/Feb/17 14:50

Description

Concurrent DDL operations may sometimes fail with a ConcurrentModificationException.
Those commands which modify the tblproperties map of a table or partition are affected, for example:

• COMPUTE STATS
• COMPUTE INCREMENTAL STATS
• DROP STATS
• ALTER TABLE SET TBPROPERTIES
• ALTER TABLE SET CACHED
• Possibly others

The logs will contain a stack trace like the one below where the exception is thrown in thrift-generated code while iterating over a HashMap.

Original Description
Recently, the stress test was modified to include COMPUTE STATS statements with a variety of MT_DOP options.

Occasionally those fail with:

I0208 03:06:26.634202 67490 jni-util.cc:169] java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
        at java.util.HashMap$EntryIterator.next(HashMap.java:962)
        at java.util.HashMap$EntryIterator.next(HashMap.java:960)
        at org.apache.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1832)
        at org.apache.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1546)
        at org.apache.impala.thrift.THdfsPartition.write(THdfsPartition.java:1389)
        at org.apache.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:1243)
        at org.apache.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:1071)
        at org.apache.impala.thrift.THdfsTable.write(THdfsTable.java:940)
        at org.apache.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1628)
        at org.apache.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1399)
        at org.apache.impala.thrift.TTable.write(TTable.java:1208)
        at org.apache.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1241)
        at org.apache.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1098)
        at org.apache.impala.thrift.TCatalogObject.write(TCatalogObject.java:938)
        at org.apache.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:487)
        at org.apache.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:421)
        at org.apache.impala.thrift.TGetAllCatalogObjectsResponse.write(TGetAllCatalogObjectsResponse.java:365)
        at org.apache.thrift.TSerializer.serialize(TSerializer.java:79)
        at org.apache.impala.service.JniCatalog.getCatalogObjects(JniCatalog.java:124)
I0208 03:06:26.653929 67490 status.cc:114] ConcurrentModificationException: null
    @          0x11c5241  (unknown)
    @          0x1597bfe  (unknown)
    @          0x11b50b3  (unknown)
    @          0x1177fbb  (unknown)
    @          0x1192c33  (unknown)
    @          0x1191ba2  (unknown)
    @          0x1190ed9  (unknown)
    @          0x118fedb  (unknown)
    @          0x13430ce  (unknown)
    @          0x15e804b  (unknown)
    @          0x15ef024  (unknown)
    @          0x15eef67  (unknown)
    @          0x15eeec2  (unknown)
    @          0x1a5661a  (unknown)
    @       0x3b81a079d1  (unknown)
    @       0x3b816e88fd  (unknown)
E0208 03:06:26.654156 67490 catalog-server.cc:282] ConcurrentModificationException: null

(To quantify occasionally, last night's run had 3 such failures out of 2031 COMPUTE STATS statements, out of 10,000 total statements.)

I haven't yet been able to figure out if there is some MT_DOP setting that makes this more likely to happen, or if we need concurrent COMPUTE STATS statements running concurrently, or what.

Issue Links

• breaks: IMPALA-4998 Table.toThrift() called without holding the table lock: test_view_compatibility_b0595633.test_hive org.apache.impala.catalog
• relates to: IMPALA-4948 Concurrent refresh against the same table fails with ConcurrentModificationException