Bulk Operation

Deprecate spark.network.remoteReadNioBufferConversion

Remove switch for remoteReadNioBufferConversion

Got fetch failed exception when new executor reused same ip address from a previously killed executor

Avoid to materialize AQE ExchangeQueryStageExec on the cancellation

percentile_approx regression since Spark 2.4

Optimize shuffle reading when both sort and combine are used.

Implements OffHeapRecyclingBufferPool for ZSTD Compression Buffer Pool

Parallel Compression Support for ZSTD

Data duplication may occur when fallback to origin shuffle block

Add SSL/TLS support for RPC and Shuffle communications

Reading blocks from remote executors causes timeout issue

Executor hangs when RetryingBlockTransferor fails to initiate retry

Porting k8s PVC reuse logic to spark standalone

Client receives zero number of chunks in merge meta response which doesn't trigger fallback to unmerged blocks

Expose amount of shuffle data available on the node

Unable to deserialize broadcasted map statuses when executor decommissioned

Separate finalizeShuffleMerge Processing to Dedicated Thread Pools

When encryption is enabled on the External Shuffle Service, then processing of push meta requests throws NPE

Spark caching in the context of a single job

Add option for applications to control saving of metadata in the External Shuffle Service LevelDB

Divided by zero occurs in PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse

Fix the problem of incomplete creation of subdirectories in push merged localDir

Log shuffle data corruption diagnose cause

SPARK-33235 JsonProtocol should skip logging of push-based shuffle read metrics when push-based shuffle is disabled

Shuffle merge finalization removes the wrong finalization state from the DB

Fix wrong child call in SQLShuffleWriteMetricsReporter.decRecordsWritten()

Correctly transform the SPI services for Yarn Shuffle Service

SASL Request Retries

Relocate Netty native arm64 libs

SPARK-33235 Fallback to original shuffle block when a push-merged shuffle chunk is zero-size

Spark 3.2.1 percentlie_approx query much slower than Spark 2.4.0

Remove push-based shuffle data after query finished

limit(x,y) + 子查询 出现数据丢失和乱序问题

Add shuffle index cache expire time policy to avoid unused continuous memory consumption

Avoid creating unnecessary SLF4J Logger

Block push fails with java.lang.IllegalArgumentException: Active local dirs list has not been updated by any executor registration even when the NodeManager hasn't been restarted

Spark Shuffle Index Cache ignore the weight of index Path

SPARK-20624 Stream is corrupted Exception while fetching the blocks from fallback storage system

Optimize shuffle error handler

Fast Fail the remaining push blocks if shuffle stage finalized

SPARK-30602 Handle fallback when merged shuffle blocks are corrupted and spark.shuffle.detectCorrupt is set to true

When push-based shuffle is enabled, a stage may not complete when retried

Optimize RemoteBlockPushResolver with a memory pool

Fix a rejectedExecutionException error when push-based shuffle is enabled

Remove an expired indexFilePath from the ESS shuffleIndexCache or the PBS indexCache to save memory.

It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connection is terminated

Avoid contention and get or create clientPools quickly in the TransportClientFactory

Check the FetchShuffleBlocks message only once to improve iteration in external shuffle service

Avoid to submit task when there are no requests to push up in push-based shuffle

Repartition+Shuffle+ non deterministic function leads to bad results