[SPARK-35743] Improve Parquet vectorized reader - ASF JIRA

XML

Word

Printable

JSON

This umbrella JIRA tracks efforts to improve vectorized Parquet reader.

is a parent of

SPARK-34863 Support nested column in Spark Parquet vectorized readers

SPARK-34859 Vectorized parquet reader needs synchronization among pages for column index

relates to

SPARK-36528 Implement lazy decoding for the vectorized Parquet reader

SPARK-36527 Implement lazy materialization for the vectorized Parquet reader

SPARK-36529 Decouple CPU with IO work in vectorized Parquet reader

1.	Refactor Parquet vectorized reader to remove duplicated code paths	Resolved	Chao Sun
2.	Introduce ParquetReadState to track various states while reading a Parquet column chunk	Resolved	Chao Sun
3.	Enable vectorized read for VectorizedPlainValuesReader.readBooleans	Resolved	Kazuyuki Tanimura
4.	Combine readBatch and readIntegers in VectorizedRleValuesReader	Resolved	Chao Sun
5.	Parquet vectorized reader doesn't skip null values correctly	Resolved	Chao Sun
6.	Refactor ParquetColumnIndexSuite	Resolved	Chao Sun
7.	Remove ColumnIO once PARQUET-2050 is released in Parquet 1.13	Resolved	Pan Bingkun
8.	Support Parquet v2 data page encodings for the vectorized path	Resolved	Parth Chandra
9.	Enhance ParquetSchemaConverter to capture Parquet repetition & definition level	Resolved	Chao Sun
10.	Refactor SpecificParquetRecordReaderBase and add more coverage on vectorized Parquet decoding	Resolved	Chao Sun
11.	Support Parquet v2 data page RLE encoding (for Boolean Values) for the vectorized path	Resolved	Yang Jie
12.	Improve WritableColumnVector to better support null struct	Resolved	Unassigned
13.	Enable spark.sql.parquet.enableNestedColumnVectorizedReader on master branch by default	Resolved	Chao Sun
14.	Skipping allocating vector for repetition & definition levels when possible	Resolved	Chao Sun