[IMPALA-3571] Thrift serialization can be very inefficient

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: Impala 2.5.0
• Component/s: Distributed Exec
• Labels: scalability
• Assignee: Unassigned
• Reporter: Huaisi Xu
• Created: 18/May/16 21:39
• Updated: 21/Sep/16 16:57
• Resolved: 21/Sep/16 16:57

Description

I saw a table with deserialized text form taking 3mb disk space, but it has more than 180mb serialized memory form. I cannot attach the table here(from a customer). you can find that in CDH-37506.

statestore stores only serialized form and then send to impala. After deserialization, it can become smaller than expected so frontend is able to store large metadata after deserialization. the problem usually happens when serializing in backend or deserializing in frontend. Fine control of split update size (see IMPALA-3499), can indirectly controls how much memory frontend uses to deserialize metadata.

We need to study this more. maybe related IMPALA-3254.

anyone know why? did I miss anything?

Issue Links

• is related to: IMPALA-3494 Thrift buffer overflows when serialize more than 3355443200 bytes in impala
• is related to: IMPALA-3254 query archiving can temporarily take 1g memory for a single 500KB query