[PIG-1031] PigStorage interpreting chararray/bytearray for a tuple element inside a bag as float or double

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.5.0
• Fix Version/s: 0.7.0
• Component/s: impl
• Assignee: Daniel Dai
• Reporter: Viraj Bhat
• Created: 20/Oct/09 23:12
• Updated: 14/May/10 06:55
• Resolved: 28/Jan/10 00:26

Description

I have a data stored in a text file as:

{(4153E765)} {(AF533765)}


I try reading it using PigStorage as:

A = load 'pigstoragebroken.dat' using PigStorage() as (intersectionBag:bag{T:tuple(term:bytearray)});
dump A;

I get the following results:


({(Infinity)})
({(AF533765)}

)

The problem seems to be with the method: parseFromBytes(byte[] b) in class Utf8StorageConverter. This method uses the TextDataParser (class generated via jjt) to interpret the type of data from content, even though the schema tells it is a bytearray.

TextDataParser.jjt sample code

TOKEN :
{
...
 < DOUBLENUMBER: (["-","+"])? <FLOATINGPOINT> ( ["e","E"] ([ "-","+"])? <FLOATINGPOINT> )?>
 < FLOATNUMBER: <DOUBLENUMBER> (["f","F"])? >
...
}

I tried the following options, but it will not work as we need to call bytesToBag(byte[] b) in the Utf8StorageConverter class.

A = load 'pigstoragebroken.dat' using PigStorage() as (intersectionBag:bag{T:tuple(term)});
A = load 'pigstoragebroken.dat' using PigStorage() as (intersectionBag:bag{T:tuple(term:chararray)});

Viraj