[AVRO-3531] GenericDatumReader in multithread lead to infinite loop cause misused of IdentityHashMap - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: 1.11.0
Fix Version/s: 1.11.1
Component/s: java
Labels:
- pull-request-available

Language:
- Java

Description

Hi,

I am working on a java project that uses Kafka with Avro serialization/deserialization in an messaging platform.

In production enrionment, we meet a serious issue on the deserialization processs. The GenericDatumReader process some how get into a infinite loop status, and it is happened accationally.

When the issue happens, The thread stack is like this:

"DmqFixedRateConsumer-Thread-17" #453 daemon prio=5 os_prio=0 tid=0x00007f2ae1832800 nid=0xef49 runnable [0x00007f2a743fc000]
   java.lang.Thread.State: RUNNABLE
    at java.util.IdentityHashMap.get(IdentityHashMap.java:337)
    at org.apache.avro.generic.GenericDatumReader.getStringClass(GenericDatumReader.java:503)
    at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:454)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:191)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
    at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
    at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
    at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
    at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
    at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
    at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
    at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
    at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
    at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:291)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
    at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
    at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
    at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
    at com.xxx.xxx.xxx.xxx.xxx.XXX.deserialize(XXX.java:252)
    at com.xxx.xxx.xxx.xxx.xxx.ZZZ.deserialize(ZZZ.java:216)
    at com.xxx.xxx.xxx.xxx.xxx.SSS.processMessage(SSS.java:152)
    at com.xxx.xxx.xxx.xxx.xxx.SSS.loopProcess(SSS.java:127)
    at com.xxx.xxx.xxx.xxx.xxx.SSS$$Lambda$172/367082698.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

We create 30 threads, and all the threads are the same as above! They all get stuck in the IdentityHashMap.get() method.

Accroding to this mail 1.7.6 Slow Deserialization, the Reader is thread-safe, But actually, it seems not.

Why?

org.apache.avro.generic.GenericDatumReader#getStringClass

/**
 * Called to read strings. Subclasses may override to use a different string
 * representation. By default, this calls {@link #readString(Object,Decoder)}.
 */
protected Object readString(Object old, Schema expected, Decoder in) throws IOException {
  Class stringClass = getStringClass(expected);
  if (stringClass == String.class) {
    return in.readString();
  }
  if (stringClass == CharSequence.class) {
    return readString(old, in);
  }
  return newInstanceFromString(stringClass, in.readString());
}

private Map<Schema, Class> stringClassCache = new IdentityHashMap<>();

private Class getStringClass(Schema s) {
  Class c = stringClassCache.get(s);
  if (c == null) {
    c = findStringClass(s);
    stringClassCache.put(s, c);
  }
  return c;
}

The IdentityHashMap is not thread-safe, which is addressed by javadoc clearly! Like Hashmap infinite loop issue in multithread using, same issue happen to IdentityHashMap,too.

My question is: Can the class GenericDatumReader fix this issue and act like real thread-safe? Or we need to avoid use the single instance of GenericDatumReader in multithread?

Thanks a lot,

Xtsong.