[MAPREDUCE-3924] Partioning Failing in certain scenarios for Hadoop Streaming using Ruby

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.20.2
• Labels: hadoop map reduce ruby streaming

Environment

Linux OS
44 Node cluster
Hadoop 0.20.2-cdh3u0 is the version

• Assignee: Unassigned
• Reporter: Subir S
• Created: 25/Feb/12 16:01
• Updated: 08/May/12 18:37
• Resolved: 08/May/12 18:37

Description

We noticed a wierd scenario. The partitioning option fails in scenario below. The map reduce just aggregates values like a word count. However the key is having 2 parts.

Eg:
Input Records: (<key1|key2><tab><qty1|amt1|qty2|amt2|qty3|amt3>
1201|420 1|24.0|2|26.0|0|0.0
1200|420 1|24.0|2|26.0|0|0.0
1200|420 1|25.0|3|52.0|1|55.0
1403|400 1|25.0|3|52.0|1|55.0
1201|420 1|24.0|2|26.0|0|0.0
1201|420 1|24.0|2|26.0|0|0.0
1403|400 1|25.0|3|52.0|1|55.0

Partioning option: -k1,1 and Comparator option -k1,1n -k2,2n works fine and the output we get is expected output
1200|420 2|49.0|5|78.0|1|55.0
1201|420 3|72.0|6|78.0|0|0.0
1403|400 2|50.0|6|104.0|2|110.0

However if we use
Partioning option: -k1,1 -k2,2 and Comparator option -k1,1n -k2,2n then the output is faulty. Seems the same key does not go to same reducer in output we see duplicate records, however this does not happen to all records. A hypothetical faulty output based on above input is:
1200|420 1|24.0|2|26.0|0|0.0
1201|420 3|72.0|6|78.0|0|0.0
1403|400 2|50.0|6|104.0|2|110.0
1200|420 1|25.0|3|52.0|1|55.0

See that records with the key (1200|420) is not aggregated. This can happen only if the record does not go to the same reducer after partioning.

Any clue why this is happening? I could not understanding what is going wrong.