Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4876

OutputConsumeIterator can't handle the last buffered tuples for some Operators

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      Some Operators, such as MergeCogroup, Stream, CollectedGroup etc buffer some input records to constitute the result tuples. The last result tuples are buffered in the operator. These Operators need a flag to indicate the end of input, so that they can flush and constitute their last tuples.
      Currently, the flag 'parentPlan.endOfAllInput' is targeted for flushing the buffered tuples in MR mode. But it does not work with OutputConsumeIterator in Spark mode.

        Attachments

        1. PIG-4876.patch
          5 kB
          Xianda Ke

          Issue Links

            Activity

              People

              • Assignee:
                kexianda Xianda Ke
                Reporter:
                kexianda Xianda Ke
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: