Description
SPARK-34647 replaced the ZstdInputStream with ZstdInputStreamNoFinalizer. This meant that all usages of `CompressionCodec.compressedInputStream` would need to manually close the stream as this would no longer be handled by GC finaliser mechanism.
In SparkPlan, the result of `CompressionCodec.compressedInputStream` is wrapped in an Iterator which never calls close. This implementation needs to make use of NextIterator which allows for the closing of underlying streams.
Attachments
Issue Links
- is broken by
-
SPARK-34647 Upgrade ZSTD-JNI to 1.4.8-7 and use NoFinalizer classes
-
- Resolved
-
- links to
