[SEDONA-290] RDD Spatial Joins should follow the iterator model

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.1
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Jia Yu
• Created: 06/Jun/23 23:54
• Updated: 14/Jun/23 05:34
• Resolved: 14/Jun/23 05:34

Description

According to Github issue (https://github.com/apache/sedona/issues/849), Similar to the DynamicIndexLookupJudgement code, the prebuilt index Left/RightIndexLookupJudgements should operate on a new custom individual iterator which can loadNextBatch so an entire partition need not be read into executor memory.

Issue Links

• links to: GitHub Pull Request #851