Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-362

Limit can not push in front of ForEach with flatten

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.2.0
    • 0.2.0
    • impl
    • None

    Description

      Currently logical optimizer will push Limit in front of ForEach with flatten. It is based on the assumption that ForEach with a flatten always increase the number of records. However, this is a false assumption. In the case that there is empty bags inside input tuple, the number of output records can be 0, thus less than input records.

      We have no way to know whether there is an empty bag in the input at optimization time. So the only solution is not to push Limit in front of ForEach with flatten

      Attachments

        1. PIG-362.patch
          3 kB
          Daniel Dai
        2. PIG-362-2.patch
          4 kB
          Daniel Dai

        Activity

          People

            daijy Daniel Dai
            daijy Daniel Dai
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: