Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-24297

LLAP buffer collision causes NPE

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0.0
    • Component/s: None

      Description

      HIVE-23741 introduced an optimization so that CacheTags are not stored on buffer level, but rather on file level, as one cache tag can only relate to one file. With this change a buffer->filecache reference was introduced so that the buffer's tag can be calculated with an extra indirection i.e. buffer.filecache.tag.

      However during buffer collision in putFileData method, we don't set the filecache reference of the collided (new) buffer: https://github.com/apache/hive/commit/2e18a7408a8dd49beecad8d66bfe054b7dc474da#diff-d2ccd7cf3042845a0812a5e118f82db49253d82fc86449ffa408903bf434fb6dR309-R311

      Later this cases NPE when the new (instantly decRef'ed) buffer is evicted:

      Caused by: java.lang.NullPointerException
              at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:778)
              at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546)
              at org.apache.hadoop.hive.llap.cache.CacheContentsTracker.getTagState(CacheContentsTracker.java:129)
              at org.apache.hadoop.hive.llap.cache.CacheContentsTracker.getTagState(CacheContentsTracker.java:125)
              at org.apache.hadoop.hive.llap.cache.CacheContentsTracker.reportRemoved(CacheContentsTracker.java:109)
              at org.apache.hadoop.hive.llap.cache.CacheContentsTracker.notifyEvicted(CacheContentsTracker.java:238)
              at org.apache.hadoop.hive.llap.cache.LowLevelLrfuCachePolicy.evictSomeBlocks(LowLevelLrfuCachePolicy.java:276)
              at org.apache.hadoop.hive.llap.cache.CacheContentsTracker.evictSomeBlocks(CacheContentsTracker.java:177)
              at org.apache.hadoop.hive.llap.cache.LowLevelCacheMemoryManager.reserveMemory(LowLevelCacheMemoryManager.java:98)
              at org.apache.hadoop.hive.llap.cache.LowLevelCacheMemoryManager.reserveMemory(LowLevelCacheMemoryManager.java:65)
              at org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:323)
              at org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.allocateMultiple(EncodedReaderImpl.java:1302)
              at org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedStream(EncodedReaderImpl.java:930)
              at org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:506)
              ... 16 more 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                szita Ádám Szita
                Reporter:
                szita Ádám Szita
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m