Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4952

When hive.join.emit.interval is small, queries optimized by Correlation Optimizer may generate wrong results

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      If we have a query like this ...

      SELECT xx.key, xx.cnt, yy.key
      FROM
      (SELECT x.key as key, count(1) as cnt FROM src1 x JOIN src1 y ON (x.key = y.key) group by x.key) xx
      JOIN src yy
      ON xx.key=yy.key;
      

      After Correlation Optimizer, the operator tree in the reducer will be

           JOIN2
             |
             |
            MUX
           /   \
          /     \
         GBY     |
          |      |
        JOIN1    |
          \     /
           \   /
           DEMUX
      

      For JOIN2, the right table will arrive at this operator first. If hive.join.emit.interval is small, e.g. 1, JOIN2 will output the results even it has not got any row from the left table. The logic related hive.join.emit.interval in JoinOperator assumes that inputs will be ordered by the tag. But, if a query has been optimized by Correlation Optimizer, this assumption may not hold for those JoinOperators inside the reducer.

        Attachments

        1. replay.txt
          1 kB
          Yin Huai
        2. HIVE-4952.D11889.2.patch
          23 kB
          Phabricator
        3. HIVE-4952.D11889.1.patch
          22 kB
          Phabricator

          Activity

            People

            • Assignee:
              yhuai Yin Huai
              Reporter:
              yhuai Yin Huai
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: