Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-1641

add map joined table to distributed cache

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.7.0
    • 0.7.0
    • Query Processor
    • None
    • Split the MapJoin into 2 stage. In stage 1, generate the JDBM file for each small table. In stage 2, load the JDBM file and do the Join operation in memory

    Description

      Currently, the mappers directly read the map-joined table from HDFS, which makes it difficult to scale.
      We end up getting lots of timeouts once the number of mappers are beyond a few thousand, due to
      concurrent mappers.

      It would be good idea to put the mapped file into distributed cache and read from there instead.

      Attachments

        1. Hive-1641.patch
          194 kB
          Liyin Tang
        2. Hive-1641(3).txt
          787 kB
          Liyin Tang
        3. Hive-1641(4).patch
          854 kB
          Liyin Tang
        4. Hive-1641(5).patch
          856 kB
          Liyin Tang
        5. Hive-1641(6).patch
          3 kB
          Liyin Tang

        Issue Links

          Activity

            People

              liyin Liyin Tang
              namit Namit Jain
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: