[SPARK-38329] High I/O wait when Spark Structured Streaming checkpoint changed to EFS

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 3.2.1
• Component/s: EC2, Input/Output, PySpark, Structured Streaming

Language

Python

• Assignee: Unassigned
• Reporter: Neven Jovic
• Created: 25/Feb/22 13:45
• Updated: 12/Dec/22 18:10
• Resolved: 16/Apr/22 20:57

Description

I'm currently running spark structured streaming application written in python(pyspark) where my source is kafka topic and sink i mongodb. I changed my checkpoint to Amazon EFS, which is distributed on all spark workers and after that I got increased I/o wait, averaging 8%

 

Currently I have 6000 messages coming to kafka every second, and I get every once in a while a WARN message:

22/02/25 13:12:31 WARN HDFSBackedStateStoreProvider: Error cleaning up files for HDFSStateStoreProvider[id = (op=0,part=90),dir = file:/mnt/efs_max_io/spark/state/0/90] java.lang.NumberFormatException: For input string: ""

I'm not quite sure if that message has anything to do with high I/O wait and is this behavior expected, or something to be concerned about?