[GEODE-120] RDD.saveToGemfire() can not handle big dataset (1M entries per partition)

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0-incubating
• Fix Version/s: 1.0.0-incubating.M1
• Component/s: core, extensions
• Assignee: Qihong Chen
• Reporter: Qihong Chen
• Created: 15/Jul/15 17:13
• Updated: 15/Jan/16 06:14
• Resolved: 17/Jul/15 21:32

Description

the connector use single region.putAll() call to save each RDD partition. But putAll() doesn't handle big dataset well (such as 1M record). Need to split the dataset into smaller chunks, and invoke putAll() for each chunk.