[SPARK-17762] invokeJava fails when serialized argument list is larger than INT_MAX (2,147,483,647) bytes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.0.0
• Component/s: SparkR
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Hossein Falaki
• Created: 03/Oct/16 04:30
• Updated: 21/May/19 04:13
• Resolved: 21/May/19 04:13

Description

We call writeBin within writeRaw which is called from invokeJava on the serialized arguments list. Unfortunately, writeBin has a hard-coded limit set to R_LEN_T_MAX (which is itself set to INT_MAX in base).

To work around it, we can check for this case and serialize the batch in multiple parts.

Issue Links

• links to: [Github] Pull Request #15328 (falaki)