Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-3952

merge map-job followed by map-reduce job

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Consider the query like:

      select count(*)FROM
      ( select idOne, idTwo, value FROM
      bigTable
      JOIN
      smallTableOne on (bigTable.idOne = smallTableOne.idOne)
      ) firstjoin
      JOIN
      smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo);

      where smallTableOne and smallTableTwo are smaller than hive.auto.convert.join.noconditionaltask.size and
      hive.auto.convert.join.noconditionaltask is set to true.

      The joins are collapsed into mapjoins, and it leads to a map-only job
      (for the map-joins) followed by a map-reduce job (for the group by).
      Ideally, the map-only job should be merged with the following map-reduce job.

        Attachments

        1. HIVE-3952-20130428-branch-0.11-bugfix.txt
          0.7 kB
          Vinod Kumar Vavilapalli
        2. HIVE-3952-20130428-branch-0.11-v2.txt
          90 kB
          Vinod Kumar Vavilapalli
        3. HIVE-3952-20130428-branch-0.11.txt
          90 kB
          Vinod Kumar Vavilapalli
        4. hive.3952.1.patch
          74 kB
          Namit Jain
        5. HIVE-3952-20130424.txt
          89 kB
          Vinod Kumar Vavilapalli
        6. HIVE-3952-20130421.txt
          27 kB
          Vinod Kumar Vavilapalli
        7. HIVE-3952-20130301.txt
          26 kB
          Vinod Kumar Vavilapalli
        8. HIVE-3952-20130227.1.txt
          27 kB
          Vinod Kumar Vavilapalli
        9. HIVE-3952-20130226.txt
          28 kB
          Vinod Kumar Vavilapalli

          Issue Links

            Activity

              People

              • Assignee:
                vinodkv Vinod Kumar Vavilapalli
                Reporter:
                namit Namit Jain
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: