Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38273

decodeUnsafeRows's iterators should close underlying input streams

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0, 3.2.1
    • 3.3.0, 3.2.2
    • SQL
    • None

    Description

      SPARK-34647 replaced the ZstdInputStream with ZstdInputStreamNoFinalizer. This meant that all usages of `CompressionCodec.compressedInputStream` would need to manually close the stream as this would no longer be handled by GC finaliser mechanism.

      In SparkPlan, the result of `CompressionCodec.compressedInputStream` is wrapped in an Iterator which never calls close. This implementation needs to make use of NextIterator which allows for the closing of underlying streams.

      Attachments

        Issue Links

          Activity

            People

              kevin.w.sewell Kevin Sewell
              kevin.w.sewell Kevin Sewell
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: