[ATLAS-2926] ZipSink: Very Large Entities Cause Out Of Memory Exception

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.2, 0.8.3, trunk
• Fix Version/s: 0.8.3, 1.2.0, 2.0.0
• Component/s: atlas-core
• Assignee: Ashutosh Mestry
• Reporter: Ashutosh Mestry
• Due: 19/Oct/18
• Created: 19/Oct/18 05:19
• Updated: 04/Aug/21 13:40
• Resolved: 22/Oct/18 16:19

Description

Steps to Duplicate

1. Setup Atlas with very large data. Atlas one entity should be about 300 MB in size.
2. Perform export with parameters that will include the large entity.

Following error will be encountered:

java.lang.OutOfMemoryError: Requested array size exceeds VM limit     at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)     at java.lang.StringCoding.encode(StringCoding.java:344)     at java.lang.StringCoding.encode(StringCoding.java:387)     at java.lang.String.getBytes(String.java:958)     at org.apache.atlas.repository.impexp.ZipSink.addToZipStream(ZipSink.java:106)     at org.apache.atlas.repository.impexp.ZipSink.saveToZip(ZipSink.java:95)     at org.apache.atlas.repository.impexp.ZipSink.add(ZipSink.java:55)     at org.apache.atlas.repository.impexp.ExportService.addEntity(ExportService.java:467)

Additional Information

During conversion of entity string to bytes, a known JDK error is encountered.