Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-1641

add map joined table to distributed cache

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.7.0
    • 0.7.0
    • Query Processor
    • None
    • Split the MapJoin into 2 stage. In stage 1, generate the JDBM file for each small table. In stage 2, load the JDBM file and do the Join operation in memory

    Description

      Currently, the mappers directly read the map-joined table from HDFS, which makes it difficult to scale.
      We end up getting lots of timeouts once the number of mappers are beyond a few thousand, due to
      concurrent mappers.

      It would be good idea to put the mapped file into distributed cache and read from there instead.

      Attachments

        1. Hive-1641.patch
          194 kB
          Liyin Tang
        2. Hive-1641(3).txt
          787 kB
          Liyin Tang
        3. Hive-1641(4).patch
          854 kB
          Liyin Tang
        4. Hive-1641(5).patch
          856 kB
          Liyin Tang
        5. Hive-1641(6).patch
          3 kB
          Liyin Tang

        Issue Links

          incorporates

          Bug - A problem which impairs or prevents the functions of the product. HIVE-1722 The result of the test case mapjoin1.q is not correct

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-1723 The result of left semi join is not correct

          • Major - Major loss of function.
          • Resolved
          There are no Sub-Tasks for this issue.

          Activity

            People

              liyin Liyin Tang
              namit Namit Jain
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: