Pig
  1. Pig
  2. PIG-1475

Large number of FILTER matching clauses causes internal error 2998

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.6.0
    • Fix Version/s: None
    • Component/s: grunt
    • Labels:
      None
    • Environment:

      Amazon Elastic MapReduce with Pig 0.6 and Hadoop 0.20.2

    • Tags:
      filter

      Description

      I'm generating reports using Pig where I only want to report on rows matching a set of regular expressions, but those regular expressions are pretty numerous.

      Pig fails with this internal error when I run FILTER with 500 terms:

      2010-06-29 15:46:04,579 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. null

      This only occurred when I ran my Pig script against Hadoop with the full dataset. When I ran Pig in local mode, with a smaller sample file, Pig handled the FILTER command just fine.

      The workaround has been to split my list into two separate lists of 250 then UNION the results, but I assume this is something that could be addressed in the code.

        Activity

          People

          • Assignee:
            Unassigned
            Reporter:
            Mike Subelsky
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development