[SPARK-28025] HDFSBackedStateStoreProvider should not leak .crc files

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.3
• Fix Version/s: 2.4.4, 3.0.0
• Component/s: Structured Streaming

Environment

Spark 2.4.3

Kubernetes 1.11 (OpenShift)

StateStore storage on a mounted PVC. Viewed as a local filesystem by the `FileContextBasedCheckpointFileManager` : 

scala> glusterfm.isLocal
res17: Boolean = true

• Assignee: Jungtaek Lim
• Reporter: Gerard Maas
• Created: 12/Jun/19 10:39
• Updated: 08/Oct/20 08:12
• Resolved: 23/Aug/19 06:12

Description

The HDFSBackedStateStoreProvider when using the default CheckpointFileManager is leaving '.crc' files behind. There's a .crc file created for each `atomicFile` operation of the CheckpointFileManager.

Over time, the number of files becomes very large. It makes the state store file system constantly increase in size and, in our case, deteriorates the file system performance.

Here's a sample of one of our spark storage volumes after 2 days of execution (4 stateful streaming jobs, each on a different sub-dir):

1. Total files in PVC (used for checkpoints and state store)
   $find . | wc -l
   431796
   
   # .crc files
   $find . -name "*.crc" | wc -l
   418053

With each .crc file taking one storage block, the used storage runs into the GBs of data.

These jobs are running on Kubernetes. Our shared storage provider, GlusterFS, shows serious performance deterioration with this large number of files:

DEBUG HDFSBackedStateStoreProvider: fetchFiles() took 29164ms

 

Issue Links

• causes: SPARK-28712 spark structured stream with kafka don't really delete temp files in spark standalone cluster
• is related to: SPARK-17475 HDFSMetadataLog should not leak CRC files
• links to: GitHub Pull Request #25488
• links to: GitHub Pull Request #25565