Pig
  1. Pig
  2. PIG-3446 Umbrella jira for Pig on Tez
  3. PIG-3767

Work with TEZ-668 which allows starting and closing of inputs and outputs

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      From Bikas Saha:

      https://issues.apache.org/jira/browse/TEZ-668 is a breaking change in TEZ trunk

      This adds a start method to the Input/Output and the processor is expected to call input.start()/output.start() for the input/output to actually start fetching/writing data. After this get committed, Hive and Pig Processors need to call start() on the input/output that they want to start. The processors may decide to not call start for an input they do not want to read (e.g. data already in ObjectRegistry) or they may choose to stagger the inputs in a certain order based on memory or processing requirements.

      1. PIG-3767-1.patch
        22 kB
        Rohini Palaniswamy
      2. PIG-3767-2.patch
        23 kB
        Rohini Palaniswamy
      3. PIG-3767-3.patch
        23 kB
        Rohini Palaniswamy

        Activity

        Hide
        Rohini Palaniswamy added a comment -

        Attaching final patch that changes guava-tez.version to guava-hadoop2.version.

        Committed to tez-branch. Thanks for the review Daniel

        Show
        Rohini Palaniswamy added a comment - Attaching final patch that changes guava-tez.version to guava-hadoop2.version. Committed to tez-branch. Thanks for the review Daniel
        Hide
        Rohini Palaniswamy added a comment -

        TEZ-688 is committed to tez trunk. Attached patch tested with that. https://reviews.apache.org/r/18203/

        Show
        Rohini Palaniswamy added a comment - TEZ-688 is committed to tez trunk. Attached patch tested with that. https://reviews.apache.org/r/18203/
        Hide
        Rohini Palaniswamy added a comment -

        Initial patch. TEZ-668 is not committed yet. Facing some issues with it where output directory is empty even though records are written to MROutput and commit is called. Will check with Siddharth Seth on that.

        Show
        Rohini Palaniswamy added a comment - Initial patch. TEZ-668 is not committed yet. Facing some issues with it where output directory is empty even though records are written to MROutput and commit is called. Will check with Siddharth Seth on that.

          People

          • Assignee:
            Rohini Palaniswamy
            Reporter:
            Rohini Palaniswamy
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development