[AVRO-3531] GenericDatumReader in multithread lead to infinite loop cause misused of IdentityHashMap

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.11.1
• Component/s: java
• Labels: pull-request-available

Language

Java

• Assignee: Christophe Le Saec
• Reporter: tansion
• Created: 09/Jun/22 03:48
• Updated: 09/Apr/24 12:26
• Resolved: 04/Jul/22 16:40

Description

Hi, 

I am working on a java project that uses Kafka with Avro serialization/deserialization in an messaging platform.

In production enrionment, we meet a serious issue on the deserialization processs. The GenericDatumReader process some how get into a infinite loop status, and it is happened accationally.

When the issue happens, The thread stack is like this:

 

"DmqFixedRateConsumer-Thread-17" #453 daemon prio=5 os_prio=0 tid=0x00007f2ae1832800 nid=0xef49 runnable [0x00007f2a743fc000]
   java.lang.Thread.State: RUNNABLE
    at java.util.IdentityHashMap.get(IdentityHashMap.java:337)
    at org.apache.avro.generic.GenericDatumReader.getStringClass(GenericDatumReader.java:503)
    at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:454)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:191)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
    at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
    at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
    at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
    at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
    at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
    at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
    at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
    at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
    at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
    at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
    at com.xxx.xxx.xxx.xxx.xxx.XXX.deserialize(XXX.java:252)
    at com.xxx.xxx.xxx.xxx.xxx.ZZZ.deserialize(ZZZ.java:216)
    at com.xxx.xxx.xxx.xxx.xxx.SSS.processMessage(SSS.java:152)
    at com.xxx.xxx.xxx.xxx.xxx.SSS.loopProcess(SSS.java:127)
    at com.xxx.xxx.xxx.xxx.xxx.SSS$$Lambda$172/367082698.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748) 

We create 30 threads, and all the threads are the same as above! They all get stuck in the IdentityHashMap.get() method.

 

Accroding to this mail 1.7.6 Slow Deserialization, the Reader is thread-safe,  But actually, it seems not.

Why?

org.apache.avro.generic.GenericDatumReader#getStringClass

 

/**
 * Called to read strings. Subclasses may override to use a different string
 * representation. By default, this calls {@link #readString(Object,Decoder)}.
 */
protected Object readString(Object old, Schema expected, Decoder in) throws IOException {
  Class stringClass = getStringClass(expected);
  if (stringClass == String.class) {
    return in.readString();
  }
  if (stringClass == CharSequence.class) {
    return readString(old, in);
  }
  return newInstanceFromString(stringClass, in.readString());
}

private Map<Schema, Class> stringClassCache = new IdentityHashMap<>();

private Class getStringClass(Schema s) {
  Class c = stringClassCache.get(s);
  if (c == null) {
    c = findStringClass(s);
    stringClassCache.put(s, c);
  }
  return c;
}
 

The IdentityHashMap is not thread-safe, which is addressed by javadoc clearly! Like Hashmap infinite loop issue in multithread using, same issue happen to IdentityHashMap,too.

My question is: Can the class GenericDatumReader fix this issue and act like  real thread-safe? Or we need to avoid use the single instance of GenericDatumReader in multithread?

Thanks a lot,

 Xtsong.

 

Issue Links

• relates to: AVRO-3903 Avro 1.11.x read getting stuck due to MapUtil.computeIfAbsent
• links to: GitHub Pull Request #1719