Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-1641

add map joined table to distributed cache

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.7.0
    • Component/s: Query Processor
    • Labels:
      None
    • Release Note:
      Split the MapJoin into 2 stage. In stage 1, generate the JDBM file for each small table. In stage 2, load the JDBM file and do the Join operation in memory

      Description

      Currently, the mappers directly read the map-joined table from HDFS, which makes it difficult to scale.
      We end up getting lots of timeouts once the number of mappers are beyond a few thousand, due to
      concurrent mappers.

      It would be good idea to put the mapped file into distributed cache and read from there instead.

        Attachments

        1. Hive-1641.patch
          194 kB
          Liyin Tang
        2. Hive-1641(3).txt
          787 kB
          Liyin Tang
        3. Hive-1641(4).patch
          854 kB
          Liyin Tang
        4. Hive-1641(5).patch
          856 kB
          Liyin Tang
        5. Hive-1641(6).patch
          3 kB
          Liyin Tang

          Issue Links

            Activity

              People

              • Assignee:
                liyin Liyin Tang
                Reporter:
                namit Namit Jain
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: