[SPARK-37710] Add detailed log message for java.io.IOException occurring on Kryo flow

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 3.3.0
• Component/s: Spark Core
• Assignee: Eren Avsarogullari
• Reporter: Eren Avsarogullari
• Created: 21/Dec/21 21:20
• Updated: 06/Jan/22 23:06
• Resolved: 06/Jan/22 23:06

Description

Input/output error usually points environmental issues such as disk read/write failures due to disk corruption, network access failures etc. This PR aims to be added detailed error message to catch this kind of environmental cases occurring on problematic BlockManager and logs with BlockManager hostname, blockId and blockPath details.
Following stack-trace occurred on disk corruption:

com.esotericsoftware.kryo.KryoException: java.io.IOException: Input/output error
Serialization trace:
buffers (org.apache.spark.sql.execution.columnar.DefaultCachedBatch)
    at com.esotericsoftware.kryo.io.Input.fill(Input.java:166)
    at com.esotericsoftware.kryo.io.Input.require(Input.java:196)
    at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:346)
    at com.esotericsoftware.kryo.io.Input.readBytes(Input.java:326)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.read(DefaultArraySerializers.java:55)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.read(DefaultArraySerializers.java:38)
    at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:789)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:381)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:302)
    at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:789)
    at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:816)
    at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:296)
    at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:168)
    at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
    at org.apache.spark.storage.BlockManager.maybeCacheDiskValuesInMemory(BlockManager.scala:1569)
    at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:877)
    at org.apache.spark.storage.BlockManager.get(BlockManager.scala:1163)
...
Caused by: java.io.IOException: Input/output error
    at java.io.FileInputStream.readBytes(Native Method)
    at java.io.FileInputStream.read(FileInputStream.java:255)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at net.jpountz.lz4.LZ4BlockInputStream.tryReadFully(LZ4BlockInputStream.java:269)
    at net.jpountz.lz4.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:280)
    at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:243)
    at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:157)
    at com.esotericsoftware.kryo.io.Input.fill(Input.java:164)
    ... 87 more 

Proposed Error Message:

java.io.IOException: Input/output error. BlockManagerId(driver, localhost, 49455, None) - blockId: test_my-block-id - blockDiskPath: /private/var/folders/kj/mccyycwn6mjdwnglw9g3k6pm0000gq/T/blockmgr-12dba181-771e-4ff9-a2bc-fa3ce6dbabfa/11/test_my-block-id 

Issue Links

• links to: [Github] Pull Request #34980 (erenavsarogullari)