Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-3952

merge map-job followed by map-reduce job

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.11.0
    • Query Processor
    • None
    • Reviewed

    Description

      Consider the query like:

      select count(*)FROM
      ( select idOne, idTwo, value FROM
      bigTable
      JOIN
      smallTableOne on (bigTable.idOne = smallTableOne.idOne)
      ) firstjoin
      JOIN
      smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo);

      where smallTableOne and smallTableTwo are smaller than hive.auto.convert.join.noconditionaltask.size and
      hive.auto.convert.join.noconditionaltask is set to true.

      The joins are collapsed into mapjoins, and it leads to a map-only job
      (for the map-joins) followed by a map-reduce job (for the group by).
      Ideally, the map-only job should be merged with the following map-reduce job.

      Attachments

        1. hive.3952.1.patch
          74 kB
          Namit Jain
        2. HIVE-3952-20130226.txt
          28 kB
          Vinod Kumar Vavilapalli
        3. HIVE-3952-20130227.1.txt
          27 kB
          Vinod Kumar Vavilapalli
        4. HIVE-3952-20130301.txt
          26 kB
          Vinod Kumar Vavilapalli
        5. HIVE-3952-20130421.txt
          27 kB
          Vinod Kumar Vavilapalli
        6. HIVE-3952-20130424.txt
          89 kB
          Vinod Kumar Vavilapalli
        7. HIVE-3952-20130428-branch-0.11.txt
          90 kB
          Vinod Kumar Vavilapalli
        8. HIVE-3952-20130428-branch-0.11-bugfix.txt
          0.7 kB
          Vinod Kumar Vavilapalli
        9. HIVE-3952-20130428-branch-0.11-v2.txt
          90 kB
          Vinod Kumar Vavilapalli

        Issue Links

          Activity

            People

              vinodkv Vinod Kumar Vavilapalli
              namit Namit Jain
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: