[PIG-1475] Large number of FILTER matching clauses causes internal error 2998

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.6.0
• Component/s: grunt

Environment

Amazon Elastic MapReduce with Pig 0.6 and Hadoop 0.20.2

• Tags: filter
• Assignee: Unassigned
• Reporter: Mike Subelsky
• Created: 29/Jun/10 19:37
• Updated: 30/Jun/10 02:48

Description

I'm generating reports using Pig where I only want to report on rows matching a set of regular expressions, but those regular expressions are pretty numerous.

Pig fails with this internal error when I run FILTER with 500 terms:

2010-06-29 15:46:04,579 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. null

This only occurred when I ran my Pig script against Hadoop with the full dataset. When I ran Pig in local mode, with a smaller sample file, Pig handled the FILTER command just fine.

The workaround has been to split my list into two separate lists of 250 then UNION the results, but I assume this is something that could be addressed in the code.