[PIG-2259] Black hole of multiple level dereference on "bag in bag" structure: cannot reach deeper levels

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Component/s: parser
• Labels: bag dereference pig

Environment

Pig 0.9.0 local version, on Linux x86 and Mac OS X 10.7.1

• Tags: dereference, bag
• Assignee: Unassigned
• Reporter: JArod Wen
• Created: 01/Sep/11 01:27
• Updated: 20/Dec/11 20:13

Description

I noticed that dereference cannot reach the second level of bag in a "bag in bag" structure. Here is a example:

For the following scripts:

a = load 'grade.dat' as (name, age, gpa);
b = load 'rate.dat' as (state, age, rate);
ag = group a by (name, age);
c = cogroup ag by group.age, b by age;
cf = foreach c generate $1.$0;

The relation c has the schema as:

bytearray, bag{tuple(tuple(bytearray, bytearray), bag

{tuple(bytearray, bytearray, bytearray)})}, bag{tuple(bytearray, bytearray, bytearray)}

so for c, $1.$0 means the first field of the bag "ag", which will be the tuple group(name, age). However after this, $1.$0.$0 and $1.$0.$0.$0 keep the same tuple but no deeper dereference. Actually we can add arbitrary number of ".$0" after $1.$0 but keep stay at the same position.

The reason for this interesting "black hole" of the dereference is when we dereferencing a bag, we automatically create another bag structure, so after we obtain the "group(name, age)" tuple from the bag "ag", a bag wrapper is added onto the tuple so it becomes

bag

{tuple(tuple(bytearray, bytearray))}

Then no matter how many dereferences are appended, this structure cannot be changed since every dereference just "takes off" the outer bag wrapper and "puts on" the same bag wrapper.

For the same reason, the following script can also produce the same "black hole":

cf = foreach c generate $1.$1.$0. ... (arbitrary number of ".$0")