[PIG-2691] Duplicate TOKENIZE schema

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11
• Labels: simple
• Hadoop Flags: Incompatible change

Release Note

TOKENIZE: the default name of the field in the schema produced by this UDF now depends on the input field. This change could break your script if you were relying on the field being called "bag_of_tokenTuples" (i.e. you were not using an AS clause to rename the field).

• Assignee: Jie Li
• Reporter: Gianmarco De Francisci Morales
• Created: 09/May/12 18:21
• Updated: 22/Feb/13 04:54
• Resolved: 31/May/12 15:02

Description

TOKENIZE produces a fixed named schema that results in duplicates if used more than once in the same generate statement.
We could paramenterize the schema on the name of the field being tokenized.

grunt> q = LOAD 'file' AS (source:chararray, target:chararray);
grunt> e = FOREACH q GENERATE TOKENIZE(source), TOKENIZE(target);
2012-05-09 20:18:37,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1108: 
<line 2, column 14> Duplicate schema alias: bag_of_tokenTuples
grunt> e = FOREACH q GENERATE TOKENIZE(source) as s_entities, TOKENIZE(target) as t_entities;
grunt> describe e
e: {s_entities: {tuple_of_tokens: (token: chararray)},t_entities: {tuple_of_tokens: (token: chararray)}}