Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-10128

BytesBytesMultiHashMap does not allow concurrent read-only access

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • llap, 1.2.0
    • None
    • None
    • Hide
      HIVE-10128: BytesBytesMultiHashMap does not allow concurrent read-only access (Sergey Shelukhin, reviewed by Gunther Hagleitner)
      Show
      HIVE-10128 : BytesBytesMultiHashMap does not allow concurrent read-only access (Sergey Shelukhin, reviewed by Gunther Hagleitner)

    Description

      The multi-threaded performance takes a serious hit when LLAP shares hashtables between the probe threads running in parallel.

      This is an explicit synchronized block inside ReusableRowContainer which triggers this particular pattern.

      Looking deeper into the code, the synchronization seems to be caused due to the fact that WriteBuffers.setReadPoint modifies the otherwise read-only hashtable.

      To generate this sort of result, run LLAP at a WARN log-level, to avoid all the log synchronization that otherwise affects the thread sync.

      Attachments

        1. HIVE-10128.patch
          31 kB
          Sergey Shelukhin
        2. HIVE-10128.03.patch
          29 kB
          Sergey Shelukhin
        3. HIVE-10128.02.patch
          29 kB
          Sergey Shelukhin
        4. HIVE-10128.01.patch
          28 kB
          Sergey Shelukhin
        5. hashmap-sync-source.png
          128 kB
          Gopal Vijayaraghavan
        6. hashmap-sync.png
          29 kB
          Gopal Vijayaraghavan
        7. hashmap-after.png
          5 kB
          Gopal Vijayaraghavan

        Issue Links

          duplicates

          Sub-task - The sub-task of the issue HIVE-10112 LLAP: query 17 tasks fail due to mapjoin issue

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-10219 BytesBytesMultiHashMap::clear() can't be concurrently used with reads

          • Major - Major loss of function.
          • Closed
          supercedes

          Sub-task - The sub-task of the issue HIVE-10112 LLAP: query 17 tasks fail due to mapjoin issue

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              sershe Sergey Shelukhin
              gopalv Gopal Vijayaraghavan
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: