Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: tez-branch
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None

      Description

      Currently, DISTINCT is implemented in a straightforward manner per https://issues.apache.org/jira/browse/PIG-3538.

      However, we can implement two types of combiner optimizations for DISTINCT, just as the MRCompiler does for map-reduce:
      1. A simple DistinctCombiner that throws away the duplicate tuples
      2. An optimizer that transforms certain uses of DISTINCT into an algebraic udf form

        Activity

        Alex Bain created issue -
        Alex Bain made changes -
        Field Original Value New Value
        Fix Version/s tez-branch [ 12324968 ]
        Hide
        Alex Bain added a comment -
        Show
        Alex Bain added a comment - ReviewBoard posted at https://reviews.apache.org/r/16717/
        Alex Bain made changes -
        Attachment PIG-3562-0.patch [ 12621905 ]
        Alex Bain made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Cheolsoo Park added a comment -

        +1.

        Committed to tez branch. Thank you Alex!

        Show
        Cheolsoo Park added a comment - +1. Committed to tez branch. Thank you Alex!
        Cheolsoo Park made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        63d 6h 14m 1 Alex Bain 08/Jan/14 01:49
        Patch Available Patch Available Resolved Resolved
        21h 50m 1 Cheolsoo Park 08/Jan/14 23:40
        Resolved Resolved Closed Closed
        316d 6h 19m 1 Daniel Dai 21/Nov/14 05:59

          People

          • Assignee:
            Alex Bain
            Reporter:
            Alex Bain
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development