[IGNITE-12543] When put List<List<SomeObject>>, the data was increased much larger.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6
• Fix Version/s: 2.9
• Component/s: thin client
• Flags: Important
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Unassigned
• Reporter: LEE PYUNG BEOM
• Created: 15/Jan/20 10:42
• Updated: 06/Aug/21 14:42
• Resolved: 12/May/20 07:03

Description

I use Ignite 2.6 version of Java Thin Client.

 

When I put data in the form List<List<SomeObject>>, 

The size of the original 200KB data was increased to 50MB when inquired by Ignite servers.

On the Heap Dump, the list element was repeatedly accumulated, increasing the data size.

 

When I checked org.apacheignite.internal.binary.BinaryWriterExImpl.java doWriteBinaryObject() method,

// org.apacheignite.internal.binary.BinaryWriterExImpl.java

    public void doWriteBinaryObject(@Nullable BinaryObjectImpl po) {
        if (po == null)
            out.writeByte(GridBinaryMarshaller.NULL);
        else {
            byte[] poArr = po.array();
            out.unsafeEnsure(1 + 4 + poArr.length +4);
            out.unsafeWriteByte(GridBinaryMarshaller.BINARY_OBJ);
            out.unsafeWriteInt(poArr.length);
            out.writeByteArray(poArr);
            out.unsafeWriteInt(po.start());
        }
    }

 

The current Ignite implementation for storing data in the form List<List<Some_Objectject>> is:

In the Marshalling stage, for example, data the size of List(5 members)<List(10 members)<Some_Object(size:200 KB)> is:

As many as 10*5 of the list's elements are duplicated.

If the above data contains five objects of 200KB size, ten by one,

50 iterations are stored and 200K*10**5 = 100MB of data is used for cache and transfer.

As a result of this increase in data size, it is confirmed that the failure of OOM, GC, etc. is caused by occupying Heap memory.

Unnecessarily redundant data is used for cache storage and network transport.

When looking up cache data, only some of the data at the top is read based on file location information from the entire data, so that normal data is retrieved.

The way we're implemented today is safe from basic behavior, but we're wasting memory and network unnecessarily using inefficient algorithms

This can have very serious consequences. Please check.

 

 

Issue Links

• is fixed by: IGNITE-12827 OutOfMemory exception when calling grid service from .NET with user type array parameter
• relates to: IGNITE-12625 Thin client: Marshaling/unmarshaling of objects performed twice
• relates to: IGNITE-12827 OutOfMemory exception when calling grid service from .NET with user type array parameter
• links to: GitHub Pull Request #7403