[SPARK-33200] Validate that all shuffle files are migrated during E2E testing - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Add vote

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

Delete

XML

Word

Printable

JSON

Details

Type: Improvement
Status: In Progress
Priority: Major
Resolution: Unresolved
Affects Version/s: 3.1.0
Fix Version/s: None
Component/s: Kubernetes, Tests
Labels:
None

Description

Currently the end to end testing only migrates a small number of shuffle files. While we can't migrate large amounts of data in the time of the integration suite we can extend the decommissioning test to migrate multiple shuffle files of different sizes.