[SPARK-48950] Corrupt data from parquet scans - ASF JIRA

Attach files

Attach Screenshot

Add vote

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: 3.5.0, 4.0.0, 3.5.1, 3.5.2
Fix Version/s: None
Component/s: Input/Output
Labels:
- correctness
Environment:

Spark 3.5.0

Running on kubernetes

Using Azure Blob storage with hierarchical namespace enabled

Target Version/s:

3.5.3

Description

Its very rare and non-deterministic but since Spark 3.5.0 we have started seeing a correctness bug in parquet scans when using the vectorized reader.

We've noticed this on double type columns where occasionally small groups (typically 10s to 100s) of rows are replaced with crazy values like `-1.29996470e+029, 3.56717569e-184, 7.23323243e+307, -1.05929677e+045, -7.60562076e+240, -3.18088886e-064, 2.89435993e-116`. I think this is the result of interpreting uniform random bits as a double type. Most of my testing has been on an array of double type column but we have also seen it on un-nested plain double type columns.

I've been testing this by adding a filter that should return zero results but will return non-zero if the parquet scan has problems. I've attached screenshots of this from the Spark UI.

I did a `git bisect` and found that the problem starts with https://github.com/apache/spark/pull/39950, but I haven't yet understood why. Its possible that this change is fine but it reveals a problem elsewhere? I did also notice https://github.com/apache/spark/pull/44853 which appears to be a different implementation of the same thing so maybe that could help.

Its not a major problem by itself but another symptom appears to be that Parquet scan tasks fail at a rate of approximately 0.03% with errors like those in the attached `example_task_errors.txt`. If I revert https://github.com/apache/spark/pull/39950 I get exactly 0 task failures on the same test.

The problem seems to be a bit dependant on how the parquet files happen to be organised on blob storage so I don't yet have a reproduce that I can share that doesn't depend on private data.

I tested on a pre-release 4.0.0 and the problem was still present.