[PIG-2537] Output from flatten with a null tuple input generating data inconsistent with the schema

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.8.0, 0.9.0
• Component/s: impl
• Assignee: Daniel Dai
• Reporter: Xuefu Zhang
• Created: 16/Feb/12 20:33
• Updated: 09/Dec/17 06:46
• Resolved: 09/Dec/17 06:46

Description

For the following pig script,

grunt> A = load 'file' as ( a : tuple( x, y, z ), b, c );
grunt> B = foreach A generate flatten( $0 ), b, c;
grunt> describe B;
B:

{a::x: bytearray,a::y: bytearray,a::z: bytearray,b: bytearray,c: bytearray}

Alias B has a clear schema.

However, on the backend, for a row if $0 happens to be null, then output tuple become something like
(null, b_value, c_value), which is obviously inconsistent with the schema. The behaviour is confirmed by pig code inspection.

This inconsistency corrupts data because of position shifts. Expected output row should be something like
(null, null, null, b_value, c_value).

Issue Links

• relates to: PIG-5201 Null handling on FLATTEN