Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4059 Pig on Spark
  3. PIG-4876

OutputConsumeIterator can't handle the last buffered tuples for some Operators

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      Some Operators, such as MergeCogroup, Stream, CollectedGroup etc buffer some input records to constitute the result tuples. The last result tuples are buffered in the operator. These Operators need a flag to indicate the end of input, so that they can flush and constitute their last tuples.
      Currently, the flag 'parentPlan.endOfAllInput' is targeted for flushing the buffered tuples in MR mode. But it does not work with OutputConsumeIterator in Spark mode.

        Attachments

        1. PIG-4876.patch
          5 kB
          Xianda Ke

        Issue Links

          Activity

            People

            • Assignee:
              kexianda Xianda Ke
              Reporter:
              kexianda Xianda Ke

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment