Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Classify whether a vertex is a map or reduce and set java.opts (mapreduce.map.java.opts or mapreduce.reduce.java.opts), memory.mb (mapreduce.map.memory.mb or mapreduce.reduce.memory.mb) and env (mapreduce.map.env or mapreduce.reduce.env) accordingly on the vertex. A simple thing would be to assume all root vertexes to be map vertexes and intermediate or leaf vertexes to be reduce vertexes.

      1. PIG-3742-3.patch
        50 kB
        Rohini Palaniswamy

        Issue Links

          Activity

          Rohini Palaniswamy created issue -
          Rohini Palaniswamy made changes -
          Field Original Value New Value
          Component/s tez [ 12321016 ]
          Rohini Palaniswamy made changes -
          Link This issue is related to PIG-3659 [ PIG-3659 ]
          Hide
          Rohini Palaniswamy added a comment -

          https://reviews.apache.org/r/17681

          Changes made:
          1) Converted the relevant MR settings to equivalent Tez settings and set them on AM, Vertex and Edge.
          2) Moved the util and helper classes (SecurityHelper and TezCompilerUtil) to a util package. Does not show up cleanly in review board. Will be doing a svn mv while committing.
          3) Fixed a issue with 1-1 edge in orderby while running pigmix where parallelism was not reflected in the second edge when the parallelism of first vertex changed after input split calculation. Also made POIdentityOutTez work with shuffle input as well when trying to test performance with 1-1 ege or shuffle edge with round robin partitioner. Shuffle edge with round robin partitioner or hash partitioner was very bad compared to MR. Even with 1-1 edge, performance is bad for L10.pig which orders by multiple columns. Still need to work on order by performance. Hoping unsorted shuffle with TEZ-661 might make it better.
          4) Created tez staging dir in FileLocalizer relativeRoot directory so that it gets cleaned up.

          Show
          Rohini Palaniswamy added a comment - https://reviews.apache.org/r/17681 Changes made: 1) Converted the relevant MR settings to equivalent Tez settings and set them on AM, Vertex and Edge. 2) Moved the util and helper classes (SecurityHelper and TezCompilerUtil) to a util package. Does not show up cleanly in review board. Will be doing a svn mv while committing. 3) Fixed a issue with 1-1 edge in orderby while running pigmix where parallelism was not reflected in the second edge when the parallelism of first vertex changed after input split calculation. Also made POIdentityOutTez work with shuffle input as well when trying to test performance with 1-1 ege or shuffle edge with round robin partitioner. Shuffle edge with round robin partitioner or hash partitioner was very bad compared to MR. Even with 1-1 edge, performance is bad for L10.pig which orders by multiple columns. Still need to work on order by performance. Hoping unsorted shuffle with TEZ-661 might make it better. 4) Created tez staging dir in FileLocalizer relativeRoot directory so that it gets cleaned up.
          Rohini Palaniswamy made changes -
          Attachment PIG-3742-3.patch [ 12627505 ]
          Rohini Palaniswamy made changes -
          Link This issue is related to TEZ-793 [ TEZ-793 ]
          Hide
          Rohini Palaniswamy added a comment -

          This patch also creates the tez staging directory under Pig temporary directory so that it gets cleaned up and we don't have to wait for TEZ-793.

          Show
          Rohini Palaniswamy added a comment - This patch also creates the tez staging directory under Pig temporary directory so that it gets cleaned up and we don't have to wait for TEZ-793 .
          Hide
          Rohini Palaniswamy added a comment -

          Committed to tez branch. Thanks Daniel for the review. TestTezCompiler is failing. But not related to this patch. Will fix that in PIG-3748.

          Show
          Rohini Palaniswamy added a comment - Committed to tez branch. Thanks Daniel for the review. TestTezCompiler is failing. But not related to this patch. Will fix that in PIG-3748 .
          Rohini Palaniswamy made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Cheolsoo Park made changes -
          Link This issue relates to PIG-3835 [ PIG-3835 ]
          Cheolsoo Park made changes -
          Comment [ Filed PIG-3835 for union followed by store. ]
          Cheolsoo Park made changes -
          Link This issue relates to PIG-3835 [ PIG-3835 ]
          Daniel Dai made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Rohini Palaniswamy
              Reporter:
              Rohini Palaniswamy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development