[BEAM-10787] External sorter 2048MB memory limit should be Hadoop only

• Type: Bug
• Status: Open
• Priority: P4
• Resolution: Unresolved
• Affects Version/s: 2.23.0
• Component/s: extensions-java-sorter
• Labels: Clarified
• Assignee: Unassigned
• Reporter: Neville Li
• Created: 21/Aug/20 15:42
• Updated: 04/Jun/22 16:37

Description

The 2048MB limit only applies when using Hadoop sequence file for external sort.
It doesn't apply when sorting is entirely in memory since we're using a List<KV<byte[], byte[]>> to buffer data and long to keep track of total bytes. Same with native external sorter.
https://github.com/apache/beam/pull/12659

Issue Links

• links to: GitHub Pull Request #12659