[SYSTEMDS-2162] Deserialization of compressed blocks w/ shared dictionary fails in Spark

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemML 1.1
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 28/Feb/18 02:30
• Updated: 28/Feb/18 04:12
• Resolved: 28/Feb/18 04:12

Description

For very large datasets that exceed aggregate cluster memory, we automatically apply compression. However, in case blocks with shared DDC1 dictionaries are evicted and subsequently deserialized this deserialization fails with mismatching number of bytes.

java.lang.IllegalStateException: unread block data
        at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2431)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)