Pig
  1. Pig
  2. PIG-3446 Umbrella jira for Pig on Tez
  3. PIG-3841

Honor Mapreduce Distributed Cache settings and localize resources in Tez

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If users ship files directly by specifiying mapred.cache.files, mapred.cache.archives using -D or set command in pig scripts, they should be added as yarn local resources before the Tez DAG is launched.

      1. PIG-3841-1.patch
        2 kB
        Rohini Palaniswamy

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          6d 2h 12m 1 Rohini Palaniswamy 02/Apr/14 23:44
          Patch Available Patch Available Resolved Resolved
          3d 17h 24m 1 Rohini Palaniswamy 06/Apr/14 17:08
          Resolved Resolved Closed Closed
          228d 13h 50m 1 Daniel Dai 21/Nov/14 05:58
          Daniel Dai made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Rohini Palaniswamy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Hide
          Rohini Palaniswamy added a comment -

          Committed to tez branch. Thanks Daniel for the review.

          Show
          Rohini Palaniswamy added a comment - Committed to tez branch. Thanks Daniel for the review.
          Hide
          Daniel Dai added a comment -

          +1

          Show
          Daniel Dai added a comment - +1
          Hide
          Rohini Palaniswamy added a comment -

          For now calling methods in classes of hadoop directly which are marked Private or Private+Unstable. If the MR code is copied to TEZ or a stable API is added to hadoop itself can switch to that.

          Show
          Rohini Palaniswamy added a comment - For now calling methods in classes of hadoop directly which are marked Private or Private+Unstable. If the MR code is copied to TEZ or a stable API is added to hadoop itself can switch to that.
          Rohini Palaniswamy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Rohini Palaniswamy [ rohini ]
          Rohini Palaniswamy made changes -
          Attachment PIG-3841-1.patch [ 12638364 ]
          Rohini Palaniswamy made changes -
          Link This issue requires TEZ-1012 [ TEZ-1012 ]
          Rohini Palaniswamy made changes -
          Component/s tez [ 12321016 ]
          Rohini Palaniswamy made changes -
          Field Original Value New Value
          Fix Version/s tez-branch [ 12324968 ]
          Rohini Palaniswamy created issue -

            People

            • Assignee:
              Rohini Palaniswamy
              Reporter:
              Rohini Palaniswamy
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development