Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4876

OutputConsumeIterator can't handle the last buffered tuples for some Operators

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • spark-branch
    • spark
    • None

    Description

      Some Operators, such as MergeCogroup, Stream, CollectedGroup etc buffer some input records to constitute the result tuples. The last result tuples are buffered in the operator. These Operators need a flag to indicate the end of input, so that they can flush and constitute their last tuples.
      Currently, the flag 'parentPlan.endOfAllInput' is targeted for flushing the buffered tuples in MR mode. But it does not work with OutputConsumeIterator in Spark mode.

      Attachments

        1. PIG-4876.patch
          5 kB
          Xianda Ke

        Issue Links

          contains

          Sub-task - The sub-task of the issue PIG-4842 Collected group doesn't work in some cases

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue PIG-4857 Last record is missing in STREAM operator

          • Major - Major loss of function.
          • Closed
          links to

          Web Link review board

          Activity

            People

              kexianda Xianda Ke
              kexianda Xianda Ke
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: