[SPARK-30242] Support reading Parquet files from Stream Buffer

• Type: Wish
• Status: Resolved
• Priority: Trivial
• Resolution: Won't Fix
• Affects Version/s: 3.0.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Jelther Oliveira Gonçalves
• Created: 12/Dec/19 22:59
• Updated: 12/Dec/22 18:10
• Resolved: 17/Dec/19 01:48

Description

Reading from a Python BufferIO a parquet is not possible using Pyspark.

Using:

 

from io import BytesIO

parquetbytes : Bytes = b'PAR...'

df = spark.read.format("parquet").load(BytesIO(parquetbytes))

Raises :

java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.String

 

Is there any chance this will be available in the future?